How do I create a thread that runs all the time in the background in a .net web site?

I intend to build a small web site that will poll a third party web service, say every 15 minutes, store the collected data in a db and display the results via web pages.

I want the polling to run 24 hours a day with or without anyone visiting the web site.

I know I could create a stand alone application that could run on the server to do this but is there a clean way to incorporate this into the website code. I need something that would be easy to deploy on a cheep 3rd party hosting site.

Any pointers in the right direction will be welcome.

Thanks

From stackoverflow

• Add a global.asax file to your website project.

  This will give you four event handlers, one of which is used when you application loads.

  You can fire off your thread here.

  Additionally, I recommend that you set the threads to background so that you can have them dissappear when you close your web app.

  System.Threading.Thread.CurrentThread.IsBackground = true;
  

  Tomalak : Would that thread be affected by automatic web server housekeeping (thread/process recycling)?

  Spence : if the process is recycled, the thread will trigger when teh application is restarted.

  Dominic Godin : Thanks for the answer. Looks like a perfect solution isn’t available when using cheep 3rd party hosting but this should work for my purpose.

• You ideally need to create a Windows Service to do it. The other suggestion of creating threads from the application startup is fine until you web server gets rebooted.

  At that point your background task won't be kicked off until the next person visits the site. With a Windows Service the background tasks will start running again as soon as the server boots.

  Creating a Windows Service using .net is fairly straight forward, certainly a lot easier than it was before .net:

  http://www.codeproject.com/KB/system/WindowsService.aspx

  You could of course fudge the other solution by getting something to run to hit your website after the server boots, but that is hacky and the "proper" way is a Windows Service.

  Edit:

  That'll teach me to read the question properly, didn't spot the requirement about working on a cheap shared host. The other solution is probably the best that you can hope for in that case as the Windows Service approach is almost certainly not possible.

  Dominic Godin : Thanks for the answer. While not applicable for my problem you still provided some very useful information.

  Greg B : Is it possible to have the Windows Service updated a setting within the ASP.NET app while both are running?

Generate sequence of integers in random order without constructing the whole list upfront

How can I generate the list of integers from 1 to N but in a random order, without ever constructing the whole list in memory?

(To be clear: Each number in the generated list must only appear once, so it must be the equivalent to creating the whole list in memory first, then shuffling.)

This has been determined to be a duplicate of this question.

From stackoverflow

• You will need at least half of the total list's memory, just to remember what you did already.

  If you are in tough memory conditions, you may try so:

  1. Keep the results generated so far in a tree, randomize the data, and insert it into the tree. If you cannot insert then generate another number and try again, etc, until the tree fills halfway.

  2. When the tree fills halfway, you inverse it: you construct a tree holding numbers that you haven't used already, then pick them in random order.

  It has some overhead for keeping the tree structure, but it may help when your pointers are considerably smaller in size than your data is.

• Oops, my question is a duplicate of Generating shuffled range using a PRNG rather than shuffling.

  Pete Kirkham : I was going to say that, but it took me ten minutes to find that post as it doesn't have very obvious keywords - so edit your question to link to it too.

• It might help to specify a language you are searching a solution for.

  You could use a dynamic list where you store your generated numbers, since you will need a reference which numbers you already created. Every time you create a new number you could check if the number is contained in the list and throw it away if it is contained and try again.

  The only possible way without such a list would be to use a number size where it is unlikely to generate a duplicate like a UUID if the algorithm is working correctly - but this doesn't guarantee that no duplicate is generated - it is just highly unlikely.

• very simple random is 1+((power(r,x)-1) mod p) will be from 1 to p for values of x from 1 to p and will be random where r and p are prime numbers and r <> p.

• Not the whole list technically, but you could use a bit mask to decide if a number has already been selected. This has a lot less storage than the number list itself.

  Set all N bits to 0, then for each desired number:

  • use one of the normal linear congruent methods to select a number from 1 to N.
  • if that number has already been used, find the next highest unused (0 bit), with wrap.
  • set that numbers bit to 1 and return it.

  That way you're guaranteed only one use per number and relatively random results.

Getting nothing while parsing XML response from the .aspx page of VS 2005 in JQuery.

Hi all,

I am not able to get xml response data from .aspx page of VS 2005. Following is the function which writes xml response on the client end:

protected void GetMailContents(double pdblMessageID) { string lstrMailContents = ""; DataSet lobjDs = new DataSet(); StringBuilder stringBuilder = new StringBuilder("");

    lobjDs = mobjCProfileAndMail.GetMailContents(pdblMessageID);

    if (lobjDs != null)
    {
         stringBuilder.Append("<Contents><From>");
         stringBuilder.Append(lobjDs.Tables[0].Rows[0]["From"].ToString());
         stringBuilder.Append("</From><To>");
         stringBuilder.Append(lobjDs.Tables[0].Rows[0]["To"].ToString());
         stringBuilder.Append("</To><Subject>");
         stringBuilder.Append(lobjDs.Tables[0].Rows[0]["Subject"].ToString());
         stringBuilder.Append("</Subject><Message>");
         stringBuilder.Append(lobjDs.Tables[0].Rows[0]["Message"].ToString());
         stringBuilder.Append("</Message></Contents>");
    }

    stringBuilder.Append("</MailContents>");

    lstrMailContents = "<?xml version=\"1.0\" encoding=\"utf-8\"?> \n ";
    lstrMailContents += stringBuilder.ToString();
    Response.ContentEncoding = Encoding.UTF8;
    Response.Write(lstrMailContents);
    Response.End();
}

Code on the client end:

    $(document).ready(function()
    {
        var varURL = document.URL;
        var varArr = varURL.split('=');
        var varMessageID = varArr[1];

        $.get("AjaxData.aspx?Mode=MODALDIALOG."+varMessageID, function(data)
        {
              $(data).find('Contents').each(function()
              {  
                 var varFrom = $(this).find('From').text();  
                 var varTo = $(this).find('To').text();  
                 var varSubject = $(this).find('Subject').text();
                 var varMessage = $(this).find('Message').text(); 
                 alert(varFrom);
             });  

        });
    });

I have written a alert for the data coming from the callback but getting nothing. If I am parsing any fixed xml then its working fine but in case getting response from the .aspx page got nothing. Is there any one who can help me out for this problem.

Thanks.

From stackoverflow

• First - writing xml through concatenation is really flakey - consider using XmlWriter / XDocument / XmlDocument instead, which will automatically escape any necessary symbols (<, &, etc) rather than result in invalid xml.

  Did you clear the response buffer before writing to it? In reality, it would be a lot simpler to do this from a raw handler (ashx) than from within the aspx page life-cycle. Or switch to MVC, which works similarly to the ashx result.

  Also - from within the jquery, you should probably speficy the type as xml - see here.

  Here's an example of a suitable handler:

      public void ProcessRequest(HttpContext context)
      {
          context.Response.ContentType = "text/xml";
          XmlDocument doc = new XmlDocument();
          XmlElement root = (XmlElement) doc.AppendChild(doc.CreateElement("Contents"));
          root.AppendChild(doc.CreateElement("From")).InnerText = "some text";
          root.AppendChild(doc.CreateElement("To")).InnerText = "some more text";
          root.AppendChild(doc.CreateElement("Subject")).InnerText = "this & that";
          root.AppendChild(doc.CreateElement("Message")).InnerText = "a < b > c";
          doc.Save(context.Response.Output);
      }
  

• First off, no need for aspx here - plain IHttpHandler will do and will also be a more natural solution.

  As for the question, make sure you clear the output stream before writing out XML and ensure that HTTP headers (specifically Content-Type) are correct. Use Fiddler to see what's going on under the hood.

Reporting services: Total and percentages for each row

I want a result to look like this (converting from a solution in Excel to reporting services):

Campaigns      Success      Not Success      Total      Success %     Not sucess %
Campaign 1     1            4                5          20%           80%
Campaign 2     4            4                8          50%           50%

How do I accomplish this task? I can add the Total column, without any problems - but i dont know how to add the sucess % and not success % columns :/

Thanks a bunch in advance! :)

/Jesper

From stackoverflow

• I think to add new columns you just need to right click and add a new column?

  For the "detail" field of the Success % column, right click and selct "expression" to get to the expression editor. Then enter the following expression

  =(Fields!Success.Value/Fields!Total.Value) * 100

  For the Not sucess column do similar except your expression would be =(Fields!NotSuccess.Value/Fields!Total.Vlaue) * 100

  (obviously the names of the actual database fields may be different in your situation)

• To add to Calanus's answer, you can format the cell to show a percent via its Right-Click Properties or by using the FORMAT function.

  =FORMAT((Fields!Success.Value/Fields!Total.Value), "P")
  =FORMAT((Fields!NotSuccess.Value/Fields!Total.Vlaue), "P")
  

  Nathan : Or you can highlight the cell in the designer and add a "p" in the Format property in the Properties window.

• If you are using a matrix then you can use the group to divide by either column or row

  Ie.

  =round((Fields!count.Value/sum(Fields!count.Value,"matrix1_RowGroup1") * 100,2) & " %"

  the field count would be the total of any given category

  Example of record set

  CAMPAIGN COUNT TYPE

  Campaing1 1 Success

  Campaing1 4 Not Success

  Campaing2 4 Success

  Campaing2 4 Not Success

Java - Distributed Programming, RMI?

I've got a doozy of a problem here. I'm aiming to build a framework to allow for the integration of different traffic simulation models. This integration is based upon the sharing of link connectivities, link costs, and vehicles between simulations.

To make a distributed simulation, I plan to have a 'coordinator' (star topology). All participating simulations simply register with it, and talk only to the coordinator. The coordinator then coordinates the execution of various tasks between each simulation.

A quick example of a distribution problem, is when one simulation is 'in charge' of certain objects, like a road. And another is 'in charge' of other roads. However, these roads are interconnected (and hence, we need synchronisation between these simulations, and need to be able to exchange data / invoke methods remotely).

I've had a look at RMI and am thinking it may be suited for this task. (To abstract out having to create an over-wire signalling discipline).

Is this sane? The issue here, is that simulation participants need to centralize some of their data storage in the 'coordinator' to ensure explicit synchronisation between simulations. Furthermore, some simulations may require components or methods from other simulations. (Hence the idea of using RMI).

My basic approach is to have the 'coordinator' run a giant RMI registry. And every simulation simply looks up everything in the registry, ensuring that the correct objects are used at each step.

Anyone have any tips for heading down this path?

From stackoverflow

• Take a look at JINI, it might be of some use to you.

  Mystic : Why do people vote down without a comment?

• Is this sane? IMHO no. And I'll tell you why. But first I'll add the disclaimer that this is a complicated topic so any answer has to be viewed as barely scratching the surface.

  First instead of repeating myself I'll point you to a summary of Java grid/cluster technologies that I wrote awhile ago. Its a mostly complete list.

  The star topology is "natural" for a "naive" (I don't mean that in a bad way) implementation because point-to-point is simple and centralizing key controller logic is also simple. It is however not fault-tolerant. It introduces scalability problems and a single bottleneck. It introduces communication inefficiences (namely the points communicate via a two-step process through the center).

  What you really want for this is probably a cluster (rather than a data/compute grid) solution and I'd suggest you look at Terracotta. Ideally you'd look at Oracle Coherence but it's no doubt expensive (compared to free). It is a fantastic product though.

  These two products can be used a number of ways but the core of both is to treat a cache like a distributed map. You put things in, you take things out and you fire off code that alters the cache. Coherence (with which I'm more familiar) in this regards scales fantastically well. These are more "server" based products though for a true cluster.

  If you're looking at a more distributed model then perhaps you should be looking at more of an SOA based approach.

  duffymo : cletus, it seems to me that the star topology you describe sounds an awful lot like an ESB, so all your criticisms apply. Would you agree?

• Well, Jini, or more specifically Javaspaces is a good place to start for a simple approach to the problem. Javaspaces lets you implement a master-worker model, where your master (coordinator in your case) writes tasks to the Javaspace, and the workers query for and process those tasks, writing the results back for the master. Since your problem is not embarrassingly parallel, and your workers need to synchronize/exchanging data, this will add some complexity to your solution.

  Using Javaspaces will add a whole lot more abstraction to your implementation that using plain RMI (which is used by the Jini framework internally as the default "wire protocol").

  Have a look at this article from sun for an intro.

  And Jan Newmarch's Jini Tutorial is a pretty good place to start learning Jini

  cletus : If you're going to go Javaspaces, personally I would only go for GigaSpaces http://www.gigaspaces.com/ (commercial though).

  duffymo : +1 on GigaSpaces. I thought Jini was a wonderful idea when I first heard about it at JavaOne in 1999, but I don't see that it's ever gotten traction. Explanations might be high licensing costs, proprietary wire protocol, etc. Feels dead as a doornail. I'm surprised by the recommendation.

  Mystic : Jini is conceptually clean and simple, you can use any wire-protocol you like and its open and free for commercial use. I'm not sure what you mean by dead, but as for popularity, I think it was bad marketing strategy from Sun when they showcased Jini as a technology for (small) devices.

  duffymo : "dead" == "I'm not aware of anybody in my small world talking or thinking about or using Jini." Admittedly my world is limited.

  Mystic : Well ok :), but calling it dead is a little inappropriate. It being a light weight framework for service-oriented computing, can only be dead if service-oriented computing is dead, because thats what Jini does. If one were to call Jini dead, RMI would be dead long before.

  duffymo : Jini as I knew it implied all Java participants. The advantage that I see with SOA, via either HTTP or XML, is that the protocols are open and endpoints can be written in either Java or .NET.

  Mystic : I think that is still true. But implementing a .Net service/client should be no biggy. Quite true, Jini is SOA, does not mandate what protocols are used. If fact you could implement Web Services on top of Jini.

• GridGain is a good alternative. They have a map/reduce implementation with "direct API support for split and aggregation" and "distributed task session". You can browse their examples and see if some of them fits with your needs.

  Kind Regards

• You may want to check out Hazelcast also. Hazelcast is an open source transactional, distributed/partitioned implementation of queue, topic, map, set, list, lock and executor service. It is super easy to work with; just add hazelcast.jar into your classpath and start coding. Almost no configuration is required.

  If you are interested in executing your Runnable, Callable tasks in a distributed fashion, then please check out Distributed Executor Service documentation at http://code.google.com/docreader/#p=hazelcast

  Hazelcast is released under Apache license and enterprise grade support is also available.

  Mystic : I never knew about Hazelcast. Was amazed with its simplicity and elegance. Wonder why you never got an up-vote. +1

• Have you considered using a message queue approach? You could use JMS to communicate/coordinate tasks and results among a set of servers/nodes. You could even use Amazon's SQS (Simple Queue Service: aws.amazon.com/sqs) and have your servers running on EC2 to allow you to scale up and down as required.

  Just my 2 cents.

  Alex Lim : Yeah, I'd agree that a message queue approach seems to be the most sound (perhaps an ESB ish solution) but the time spent setting up the infrastructure vs returns is rather low. The project im working on is really a prototype.

• Have a look at http://www.terracotta.org/

  its a distributed Java VM, so it has the advantage of being clustered application looks no different than a standard Java application.

  I have used it in applications and the speed is very impressive so far.

  Paul

• Just as an addition to the other answers which as far as I have seen all focus on grid and cloud computing, you should notice that simulation models have one unique characteristic: simulation time.

  When running distributed simulation models in parallel and synchronized then I see two options:

  • When each simulation model has its own simulation clock and event list then these should be synchronized over the network.
  • Alternatively there could be a single simulation clock and event list which will "tick the time" for all distributed (sub) models.

  The first option has been extensively researched for the High Level Architecture (HLA) see for example http://en.wikipedia.org/wiki/IEEE_1516 as a starter.

  However the second option seems more simple and with less overhead to me.

Lua COM Programming

Anyone know of any examples that shows COM programming via Lua? Could require a library.

I need to write some external scripts for Photoshop, so wondering if it's possible with Lua.

From stackoverflow

• Have a look at Lua for Windows. It contains LuaCOM. I haven't tried it myself, though.

• As gimpf said, take a look at LuaCOM. You can either get it from its official site, which is the latest version (1.4). Unzip it somewhere and take a look at the folder "Demo", which includes examples for:

  • ADO
  • Powerpoint
  • SAPI (Speech API)
  • WMI

  They are more like test cases than examples but you can get the idea.

• You can roll your own system using CInvoke and some custom C/C++ code combined with Lua meta-tables.

  http://www.nongnu.org/cinvoke/

  From experience though I can't recommend this, it is pretty complicated and requires a lot of effort to get it right and make it bulletproof.

Activation function when training a single layer perceptron

When training a multi-layer neural network, using a sigmoidal activation function is necessary for it to learn efficiently.

Is there any advantage to using a sigmoidal activation function when training a single layer perceptron, or is a simple step (heaviside) function sufficient (or even preferable)?

I'm slowly getting my head around neural networks but any help with this would be appreciated.

From stackoverflow

• Yes there is an advantage. The result can be something between 0 and 1 and doesn't have to be either YES or NO, but also MAYBE. Even for a single-neuron model it's better to have a non-step activation function.

  If you need it depends on how your output is read out. Do you need binary (YES, NO) values or also something in-between?

  I think you could also use a linear function, if you don't want to use a sigmoidal function.

• The heaviside step function is discrete, while the sigmoid is continuous, allowing you to obtain the distance from the testing point to the learnt hyperplane.