Sunday, June 17, 2018

HTTP Timeouts in Node



I needed to update some code at work the other day to add a timeout to an HTTP GET request.  Simple enough, I thought.  How hard could it be?

Looking back, it is pretty straight forward, and the code makes perfect sense, but definitely wasn’t intuitive when first trying to get it working.  I’m writing this mainly as a note to myself about the proper way to handle this in the future.

The Starting Point

The code I was updating called a GET endpoint on localhost.

var req = http.get(url, res => {
    //handle success
}).on('error', e => {
    //handle error
});

Adding a Timeout

I hadn't added a timeout to a request in Node before, using the built-in HTTP library, so I did what any red-blooded developer would do, and copy-pasted from elsewhere in the code that was doing something similar.

    var req = http.get(url, res => {
        //handle success
    }).on('error', e => {
        //handle error
    }).on('socket', sock => {
        sock.setTimeout(500);
        sock.on('timeout', () => {
            req.abort();
        });
    });

This seemed oddly confusing for something so simple, but worked fine.  Reading the docs, I could see that the 'socket' event fires when a socket is assigned to the request.  We then set a timeout directly on the socket, and abort the request once that occurs.

I thought the abort wasn't necessary, since the request had already timed out, so I removed it and it still worked.  The timeout event would fire right away, instead of waiting the default 2 minutes, like it used to, to error out.  I was actually missing something important, but hadn't noticed it yet.

HTTPS

Next, I had to update http to https, which normally just means adding a single letter to your require statement.  Since this was localhost though, and didn't have a certificate, I also needed to ignore certificate errors.  Again, I used my normal trick of finding another call that did this, and copying from it.

I found a POST call to a different localhost endpoint, and saw that it was using https.request (instead of https.get) and that it was using an options object, instead of passing the URL as a string.  The options object had a 'rejectUnauthorized' property, which is what I needed to ignore certificate errors.  Why they used the term 'Unauthorized', instead of 'InvalidCert' or something similar, is beyond me.

I figured that request() just always used an option object, instead of a URL like get() did, and to get access to the 'rejectUnauthorized' property, I would need to need to switch to it.  So, I changed my http.get to https.request, passed it an option object, breaking out the host, path, port and method, and passed that in, instead of the URL string.

let options = {
    host: 'localhost',
    path: '/resource',
    port: '12345',
    method: 'GET',
    rejectUnauthorized: false
};
var req = http.request(options, res => {
    //handle success
}).on('error', e => {
    //handle error
}).on('socket', sock => {
    sock.setTimeout(500);
    sock.on('timeout', () => { });
});

Simple enough, but it didn't work.  The call would hang.  It took some digging, but it turns out I missed an important line from the request() example I copied from.  I knew that the get() method I was using before would automatically set the method to GET, but apparently it also calls end() on the request, which I didn't realize was needed.  It makes sense, as the way request are modified by method chaining means you wouldn't know when you were done updating the request, but it wasn't obvious at first.

After I got that figured that out, I then realized that both get() and request() can take either a url or options object, so I actually didn't need to use request(), and switched back to get() (and removed the call to end() I had just added).  Everything was working now, until I pushed the code to an environment where the logging code actually wrote to somewhere.

Aborting

It turns out that whenever my request timed out, I was still logging an error, but only after 2 minutes.  Again, I dug through the docs and found that when a request times out, it doesn't actually end.  It would keep going until it was forcefully timed-out and killed, after the default 2-minute timeout.  In the timeout event, you still need to manually abort the request, which the original code I copied from did.  Yet another example of it never being a good idea to remove code you don't understand, until you understand it.

That change still didn't get everything working though.  The code would still throw an error ("socket hang up"), but it would do it immediately after the timeout, instead of after 2 minutes.  The final piece was realizing that aborting a request still throws an error, but sets the 'code' property of the error object to 'ECONNRESET'.  So when I checked for that in the error handler, I could successfully ignore timeouts, and handle regular errors.  Annoyingly though, error.code is also set to 'ECONNRESET' if the server resets the connection, so to be truly accurate, you would need to set a variable in the 'timeout' event that the 'error' event could check.  For my needs though, that wasn't necessary, and I just checked error.code.

Wrapping Up

In the end, I got the code working, and everything makes sense now.  While I understand the logic behind most design decisions of the API (the name of 'rejectUnauthorized' not withstanding), I feel like many parts aren't very intuitive, especially for those coming from a different background, such as .NET.

After finding out about request.setTimeout, which mimicks the more verbose socket code, the final code is below:

let options = {
    host: 'localhost',
    path: '/resource',
    port: '12345',
    method: 'GET',
    rejectUnauthorized: false
};
var req = https.get(options, res => {
    //handle success
}).on('error', function(err) {
    //ignore timeouts
    if (err.code === "ECONNRESET") {
        return;
    }
    //handle error
}
).setTimeout(25, () => {
    req.abort();
});

Thursday, December 21, 2017

How to Use Span


I've read plenty of articles about Span over the past year or two, and have been following its development from a distance, without actually trying any of the early iterations of it.  When Microsoft released a blog post welcoming Span, I figured it was finally time to try it out myself.


Where Is It?

Amazingly, the mentioned post above, and even the Channel 9 video C# 7.2: Understanding Span, never actually mention how to get Span compiling.  Searching for people that have actually used it, turned up nothing.

It wasn't until I read a comment on the article that I saw that you still need to reference the Nuget library System.Memory to get it to work.  Even more, the library is still in pre-release, so it is not actually officially released yet.  Based on the wording of the announcement post, this was surprising.

It appears that .NET Core 2.1 (and I assume .NET Standard 2.1) will have full support, as well as the additional library changes they keep mentioning, that have all of the overloads supporting span.

Adding Support

Anyways, the only thing you need to do, once you update to Visual Studio 2017 15.5, is to add the nuget package.

Install-Package System.Memory -Prerelease

You may still get some red squigglies with some code, as the editor doesn't have full support for it yet, but the following should compile.

var array = new int[10];
Span span = array;

Thursday, March 27, 2014

Loading Real Images to the Windows Phone Emulator

default_image_0

If you have ever used a Windows Phone, you have seen the 9 images loaded on every phone by default.  And if you are a developer, you have also seen them on the emulator, and noticed that they are the ONLY real images available without doing some tricks to get extra ones on there.

This is fine for testing, but I’ve seen too many screen shots in the store, and even promotional images and videos using them, even when the images are completely inappropriate for the app.  There is a great post by Matt Lacey about the 9 images that shouldn’t appear as part of your screen shots where he talks about this issue.

The following images were found in a few minutes by doing a quick search of the Photo section of the Windows Phone store.

wp_ss_20140326_0001 wp_ss_20140326_0002 wp_ss_20140326_0003 wp_ss_20140326_0004

Loading Real Photos

When I wrote my first Windows Phone app, Memorylage, just over a year ago, I wanted to make sure that my screen shots and promotional images really showed off my app, but wasn’t sure at first how to get any other images on there.  I could have taken screen shots directly from my phone, but that wouldn’t work for the 16:9 resolution that the 8X used.  Also, I made a promotional video doing a screen capture of the emulator, so I needed them on there for that as well.

I tried uploading some images to a web page on my site, and manually downloading them from the browser on the emulator, but that wasn’t very efficient, and I needed a lot of images to show off my app.  I needed a better way.

I ended up finding a command line program that is part of the Windows Phone SDK called IsolatedStorageExplorerTool.exe.  This lets you view the contents of an app’s isolated storage, either on a device, or on the emulator.  You can also download the contents to your computer, or even upload new content from your computer.

I had found a way to get files onto the emulator, but the program wasn’t exactly user friendly, and is limited in what it can do.  I ended up writing a simple phone app that took the images within its local storage, and saved them to the camera roll and saved pictures folder.  (These are the only two parts of a user’s photo library that an app can write to.)  I then used the .exe to load images to that app’s storage, and ran the app to get the images loaded directly into the emulator.

Doing Better

It all worked, and I was able to do all of my promotional material from the emulator without having to look at the default pictures.  I felt like the whole process could be improved though, but didn’t have time then, and left it.  Recently, however, I have been seeing the images still used in apps, even by seasoned devs, and felt it was time to take my solution, clean it up, and release it for others to use.

So, without further ado, here is the Visual Studio solution for my app.  (I am planning on hosting it on CodePlex, but was having trouble getting Git to successfully upload to it last night, so will update the post once it is there)

WPEmulatorImageLoader.zip

One Solution, Two Apps

The first project within the solution is a desktop app that allows you to select a folder containing images you want loaded to the phone’s camera roll, as well as a folder for images that will go in the saved pictures folder.  There is then a button to load those pictures to the emulator, which will use IsolatedStorageExplorerTool.exe internally.

The second project is a Windows Phone app that has two buttons for loading images from isolated storage to the camera roll and saved pictures folder on the phone/emulator. (Pardon the programmer UI)

ImageLoaderDesktopApp ImageLoaderPhoneApp

Using the applications is a 3-step process.

  1. Run the phone app within the emulator, but don’t press anything.  Just close the app immediately.  This must be done first, to load the app onto the emulator, so that the desktop program can then load data into it.
  2. Run the desktop app, and select a folder for the camera roll and saved pictures folder.  You can do one or the other, or both.  (By default, it will recursively load all sub-folders as well, but you can un-check that option)  After the folders are selected, just press the load button, and wait for the command windows that pops up to dissapear.
  3. Run the phone app again, but this time press the button for the camera roll and saved pictures folder.  These will now work, as the isolated storage should contain all of the images.

Wrap Up

And that is all you should need to get your own images, even hundreds or thousands of images, onto the emulator.  With this, there really isn’t any excuse for using the defaults anymore, so hopefully we’ll be seeing less of them in the store and on Twitter.

If anyone finds this useful, please let me know, as I would love to hear about it.

Monday, February 10, 2014

AdDuplex Pricing Update

image[13]

Most Windows Phone or Windows Store developers know about AdDuplex (which I am a Developer Evangelist for), and many use them to help promote their apps.  They do this either by adding their control to their own apps (earning free ad impressions) or by simply purchasing ad impressions directly.  For those doing the latter, their pricing just got a little better, but has a higher starting point.

You used to be able to get 20,000 impressions for $60, which was a cheap way to test them out, but didn’t give you many impressions, and wasn’t very cost-effective.  As some developers didn’t have enough money to purchase the larger packages though, they were stuck with this package, meaning they couldn’t get much for their money.

Adjusting to feedback, AdDuplex has replaced that package with a 50,000 impression package for $99.  While costing a little more, it is still cheap enough for small devs to afford, and offers a 33% better CPM (cost per thousand)!  It used to be $3, but is now $1.98.

Even better, if you currently have a published app with the AdDuplex control installed in it, they are currently giving a 20% discount on all ad packages, so be sure to check it out.

Friday, January 24, 2014

Windows Phone Dev Day - Cambridge, MA

Jump-Start Your App, or Pass the Final Stretch

If you are a developer that is trying to finish up your app, and get it into the store, or if you are looking to start your very first app, Nokia wants to help.  In just over a week, on February 1st, they will be hosting an event at the Microsoft offices in Cambridge, MA.  There will be training, food and prizes, and while seats are limited, here is what you can look forward to:

  • 12 PM DevCenter Submission Walkthrough (30 minutes): See just how easy it is to submit and publish an app worldwide
  • 12:30 PM Build an app in 30 minutes with App Studio (30 minutes): See how to build an app from an idea to publishable in 30 minutes, literally.
  • 1 PM - 1:30 PM Lunch: I’ll be providing food and snacks. This is when you think about what you need help with or when you should think of your app idea for App Studio.
  • 1:30-6 PM Hands-on time: The experts will be available to help during this time. They will be walking around helping and/or attending to requests for help.

Focus Can be the Most Important Part

While having plenty of Windows Phone experts around to answer questions, walk through the submission process, and demonstrate App Studio can be helpful, the most useful part of the day will quite possibly be the dedicated time to work on your app.  Having a solid block of time to work on your app without distraction is extremely valuable, and can really help you crank through your to-do list, especially when there are people that can help your through any road-blocks that normally might slow you down.

Links and Stuff

If you are interested in going, the link to signup is below:

http://www.eventbrite.com/e/boston-wpdev-day-tickets-10293069849

Here is also a post from Lance McCarthy (the guy running the show) going into a little more detail.

http://nokiawpdev.wordpress.com/2014/01/21/get-it-in-the-store/

I will be there helping out, and would love to see lots of people from the area show up, but as I said, seats are limited, so please don’t register unless you are sure you can make it.

Friday, January 10, 2014

Lessons From the AT&T Hackathon

Every year, AT&T hosts a hackathon and developer summit the weekend before CES starts.  This year, I decided to enter, and created a team with Anthony Russell (@Anth0nyRussell), Kevin Wolf (@ByteMaster) and Alex Perez (@Aapg1102).  We wanted to give Windows Phone a strong showing, but had a few challenges to work through.

Below are some thoughts and lessons from what we went through.

What to Write

We had a few things to consider when planning out what to write for the hackathon.  We didn’t want to waste our time writing something that was closely tied to custom hardware, and would end up being discarded after the event, but in the wearables category, it is hard to get a winning entry otherwise.  We also felt that we had a disadvantage on the hardware front, as most wearable tech is made for iOS and Android, with no support for Windows Phone. A clear strength we felt we had though, was productivity in building the actual app.  C#\MVVM and Visual Studio can fly through app development faster than xCode or Eclipse could hope to.  Adding in an Azure back-end was also an easy addition that, while available for other platforms, probably wouldn’t be used much at the competition, and could be a big differentiator.

Having said that, we looked at the hardware we had to work with (Kevin and Anthony were the two main hardware guys) and when talking about the SparkFun LEDs we had, we came up with the idea of sewing them to the back of a vest, and making turn signals out of them.  That sparked various ideas about controlling them, and got us thinking about a cycling app to go along with everything.  And trying to play up our productivity strengths, we wanted to really build out a full app, and not just some control prototype. And from that, the idea of FitBike was born...

It would be a cycling app, like many other fitness apps, but would have a social aspect as well, allowing you to create routes that you can share with others, post times that you ride them so others can join, add friends to track what they’ve been up to and where there riding, and even gamifying the experience by having badges for completing various objectives.  On top of that, the part of the app that runs while you are riding would be able to interact with multiple different pieces of hardware, including:

  • Pebble Watch – This could be updated with information about your ride performance (time, distance, avg. speed, etc) as well as alert you to upcoming turns in the route
  • Custom LED Vest – This vest was custom built with SparkFun LEDs, allowing the rider to control turn signals on their back while riding at night
  • Custom Gloves – These gloves are what control the vest, by letting the rider twist their wrist to the left or right to activate the turn signals.
  • GoPro Camera – If one was mounted to the bike, it could be controlled as well by the gloves, allowing the rider to easily take pictures along the ride without distraction.


How to Code It

Once we knew what to write, we started planning out who would be in charge of the various tasks, and how the code itself would be structured.  I can’t stress just how helpful this was to enabling us to quickly execute on our plans, as we had an ambitious scope for 24 hours. While everyone had various strengths, many of them overlapped, so it wasn’t always easy to decide who did what, but we ended up breaking the basic tasks as follows:

Kevin: Hardware setup and coding, as well as Azure setup

Anthony: Cycle route creation and tracking

Alex: Visual design, and initial screen creation

Myself: Overall app structure (allowing shared code between Win8 and WP8), data flow and Nokia Imaging SDK.

When we met in person in Vegas, we started going through the app in more detail, picking the feature list we thought we could create, and drawing out diagrams showing how the user would move through the app. This diagram was extremely helpful for assigning tasks further, and especially for Alex, who was able to just blow through creating gorgeous screens for each page in the diagram.  Also, I had come up with the general coding structure we would use within the app, and went through that with the team, so we were all in agreement with where the various different parts of the code would sit, and how they would be formatted.  (We even went over things such as how to case variables, where I was out-voted, having to change to use an underscore for private member variables, instead of just lowercasing it.)

All of these things gave us a huge head start, as we knew exactly what to do once we started, and didn’t have to waste any time with discussions.  I immediately started creating the initial solution in Visual Studio, with a project for the phone app, Windows 8 app, and a core PCL of shared code; as well as installing all required libraries such as MVVM Light (PCL), Azure Mobile Services (PCL) and the Nokia Imaging SDK (WP8 only).  Kevin started setting up everything in Azure, including any tables we would need, and creating the base classes that handled talking to Azure.  Anthony started a new phone project where he could create the code and UI for creating routes, as he knew exactly how he would need to plug it in to the main app once it was ready.  And as stated earlier, Alex was able to quickly create awesome looking XAML pages for each of the screens we had laid out earlier, and start on the various graphics we would need in the different parts of the app (logos, icons, badges, etc).  And once Kevin finished creating all of the tables, I was able to quickly create the view models that represented them, giving us a fairly stable core of the app within a couple hours of starting.

You Can’t Do It All

After that, and despite out best efforts (as with all projects), simple tasks started taking longer than planned, and schedules started to slip.  While the big points that we had discussed really came together well, the subtle parts of each area needed more discussion throughout the event.  Even worse, the simple things that we had all done before (and we therefore didn’t allot much time to) just wouldn’t work, and often took hours to fix.  Getting cropping and resizing in the Nokia Imaging SDK working with my UI code was causing me grief, storing and retrieving images in Azure BLOB storage was taking Kevin longer than planned (as well as a NuGet referencing issue), and Anthony was having trouble getting points tapped on a map to sync up properly onto the street.

With the broad scope of our project plan, there were enough little issues piling up, and we soon realized we would need to start cutting some features.  We had planned ahead for this as well, and knew what features were “nice to haves”, but not required, and started dropping those from the app.

Probably the biggest mistake we made, as far as project planning, was to save the hardware code for the end.  Since pretty much everything we were doing with hardware was just sending simple commands to controls them (which Kevin had done many times before), we assumed that would be a quick thing to integrate once everything else was up and running.  The main hardware piece (the vest and gloves) of course wouldn’t work correctly when the time came to start integrating them in.  We couldn’t figure out the cause, but even once they were working, they weren’t very reliable.  In the end, we realized that the extreme Bluetooth interference was causing most of the trouble.  The hardware wasn’t used to having hundreds of people around all running various types of Bluetooth and other wireless technologies.

One Last Surprise

The last things that caught us by surprise, and really hurt our plans, was the announcement that there would be only 90 seconds for the presentation, and they would be very strict about enforcing it.  The reason this affected us so much was that one of our initial strategies was to impress, not just with the hardware, but with how full-featured of an app we could create to go along with it.  When you only have 90 seconds to explain and show-off everything though, breadth of features is not what will help.  We ended up having to quickly fly through a summarized list of what it could do, and in the end, barely had any time to focus on the wearable side, which was what the focus of the competition was.

What Was Learned

Summing up the things that really worked were:

  • Know what you are going to write.  I have done hackathons by myself where I didn’t know what I was going to write ahead of time, but working with other means that there is a lot more communication you don’t want to be doing on the clock.  (My app LockMapper was actually created at a hackathon where I didn’t know what I wanted to do when I walked in.)
  • Sweat the details.  You may think that everyone on the team knows how to write an app, and may even all be on board with MVVM, but there are a lot of different ways to implement things, so again, at least get the basics figured out, and anything else you can think of that might be done differently by everyone (such as naming conventions, what does/doesn’t go in a view model, etc).
  • Plan out the flow of your app.  Even if you know what it will do, and roughly how it will work, actually drawing out the screens, and connecting them, on paper will help immensely in planning and execution.
  • Plan out your roles.  You don’t want two people working on the same thing, and you don’t want one person waiting on someone else.  Figure out who is building what, and in what order various pieces need to be accomplished.  That way, everyone knows what to work on while waiting for other pieces to be ready.
  • Cut things out quickly.  If there are things that aren’t core to your app, and are giving you trouble, cut them out immediately.  If there is time, you can always add them later, but if there isn’t, you can never get the time back that you sunk into them.
Here are some of the things that caused us some trouble, and what we would do differently next time:
  • Keep a narrow scope.  While I was aware of this general principal when first deciding what to write, I thought of the broader scope as a huge advantage for us if done right, but it ended up being a hindrance in the end.  As a bonus point, finding out how long you will have to present as soon as possible is also extremely important.
  • Figure out the most important aspects of your app, and do them first.  While we tried to do this, the broad scope kept us from doing it well.  We were in a wearable competition, and kept that piece until the end.  Granted, Kevin (who did most of the hardware programming) was also doing the initial Azure coding (which I would have been waiting on), we should have prioritized any wearable coding much sooner.
  • Have a presentation planned ahead of time, and decide who is going to do it.  As programmers, we naturally think that the code is the most important part of a hackathon, but it isn’t.  The winners almost always have a dead-simple idea that can be explained easily, and is easy to see its usefulness.
In the end, we didn't place in the top 3 for our category (although some judges we talked to said we were in the top 5), but ended up taking home 10K for the Nokia Windows Phone challenge, which was pretty awesome.  The competition as a whole was a ton of fun, and many of us are looking forward to returning next year, but when we do, we will be a little wiser from our experiences this first time.

Wednesday, January 8, 2014

Microsoft is Adding Carrier Billing to More Countries

…but it will cost you… but it will probably help you

Carrier billing is quickly becoming more common throughout the world as an option to pay for things on your phone.  Beyond convenience, the benefit to consumers is that those without credit cards are still able to make purchases.  This is extremely important in countries where the vast majority of the population doesn’t even have them.

Microsoft is Working on It

Microsoft knows this, and knows that people with Windows Phone that can’t pay for apps aren’t as helpful to the developer community as those who can.  Last year they tripled the number of carriers they could bill through to 53, across 53 markets, and have been adding gift cards as an additional option to even more markets.  For some markets though, this comes with a cost, as carrier billing can have a higher transaction fee than credit cards.

To help make it financially viable for them to keep expanding carrier billing to those markets, Microsoft is adding an additional 13.9% charge on top of the 30% they already charge, but only on purchases made with carrier billing in these countries.

This is Actually a Good Thing

While many people will complain about Microsoft charging developers even more, I think this is actually a great change for everyone.  As said before, this allows Microsoft to add carrier billing to countries where it wouldn’t be financially viable before, and developers get a much larger pool of paying customers in those markets.  Remember, that most of the people this fee will be charged against would not have been able to pay for ANYTHING before.  Now they finally can.  The extra money from those new paying customers will almost certainly dwarf the loss of the fee from those who would have paid regardless.  And this has no effect on the majority of markets, or anyone who still pays with a credit card.

Time will tell how much of an effect this will have, but I for one, am excited for it.