Windows 8 Doesn’t Want Your App. Try Again Later

[Update 3: And it finally passed certification and is now in the store!  (Oddly enough though, I didn't actually change any code between the last release and this one.)  Thanks to everyone that checked this post out and commented here or elsewhere, and thanks to the people at Microsoft that helped me through a lot of the issues.  While I wrote this to point out the flaws in the overall certification process, and ways it could improve, the individual people that I worked with were always wiling to go out of their way to try and help.  As mentioned below, they already had this escalated it internally before this was posted.]

[Update 2: A lot of people are commenting on various sites about how it is my fault that my app failed, as it was due to problems with my code.  I agree completely, but the point of the article is not that Microsoft failed my app, but that they don't clearly explain why they failed it, which turned me failing once or maybe twice to failing 6 times.]

[Update: I just went to the Microsoft Store this morning to pickup a Surface tablet (so I could test on an ARM computer), and after getting my pre-order slip over the weekend, and being told that they open early at 8, I show up at 7 and am first in line!  At 7:30 though, I overhear that Microsoft sent out an email yesterday saying that they couldn't open the store until their normal time of 10.  I have a job, so I left Surfaceless and un-happy.  #NotHowToLaunchAMajorProduct]

To start, let’s get one thing straight: Microsoft desperately needs apps in its store.  If not only for Windows 8, especially for Windows RT, which can only run apps from there.  It also wants as many apps in there as possible, and has been very active in courting and helping developers to write app for it.  For the most part, I think they have done a great job, as the tools, documentation and examples are excellent.  Where they really fall down though is in the last mile: app submission.

I first submitted my app, Memorylage, to the store on August 29.  Since then, it has failed validation 6 times, and I still don’t know what is causing the failure.  Below, I talk about what I have gone through in the past 10 months, trying to get my app ready, but since it is a little long winded, I have a “Short Version” further down if you want to skip to there. (I won’t blame you)  Also, please subscribe to my blog if you find it interesting.

The Long Version

Getting Ready For Submission

Memorylage has been in development since December, back when there was only the Developer Preview to use.  It was sorely lacking documentation and examples back then, but was enough to get familiar with what is a very different development experience from the Windows of old.  By Consumer Preview, things had improved greatly, especially in the templates, and Memorylage was starting to take shape.

Right around the release of the Release Preview, I attended a 3 day App Excellence lab that Microsoft hosted in their Waltham, MA office.  As this was just a hobby, I had use up my vacation time to go, but it was a great way to get some focused development done, and meet some other developers.  I also won the App X contest at the end, netting me a new Lumia 800 Windows Phone.  Best of all though, I was able to schedule time with a Microsoft Field Engineer, flying in from Redmond, to review my app for early access to the store.

The review involved a 60-point checklist we went though, which checked to make sure that the app met all aspects of their design guidelines, and really showcased how a Windows 8 app was supposed to work.  He also dug deep into the performance and stability of the app, helping me find ways to improve the responsiveness, and give me ideas about how to improve it.  There ended up being 3 issues for me to fix before he would grant me early access (2 of which I knew about before the meeting, but wanted input on), but a few weeks later (July 12th), I had resolved the issues and was granted an early access developer token.

While I had the token, allowing me to submit to the store before almost anyone else, I really wanted to make sure the app was polished and feature complete before submitting, so I kept working on it to get it ready.  I then went to a Hackathon, hosted by Microsoft at the Museum of Science in Boston, on August 18th where I won my second App X competition, netting me a Samsung Series 7 Slate, a $500 gift card and an XBox game.  I also met plenty of people from Microsoft and elsewhere who were really interested in my app, and excited to see it in the store.  They encouraged me to just hurry up and submit it, so I was really feeling good about the possibility of it taking off.

I made up a list of things I felt I need to have ready before submitting, and quickly worked through them, eventually submitting to the store on August 29th.  With only 500 apps in the store, and almost 2 months before general release of Windows 8, I was feeling pretty good about the work I had put into it.  A day later though, I got the first of what would be many failure reports.

Where Things Fall Apart

Memorylage failed a day later for 4 reasons, and with the exception of the 1st, there really wasn’t much insight into what the issue was.

• Requirement 1.2: The website linked in the app and listing page was not finished
• Requirement 3.2: The app crashes
• Requirement 3.8: Performance
• Requirement 3.10: Failed the Direct 3D test

I was surprised that they actually looked at the web site (I was planning on having it up by the time it made it to the store), but was happy to see that level of detailed review, as I figured it reflected well on the level of end-to-end quality that they were looking for. For the other failures though, I was left with quite a few questions.

• When does the app crash?
  
  • At startup?
  • During a certain task?
  • Every time, or randomly?
• What performance problems are there?
  
  • Does it startup too slow?
  • Does it suspend too slow?
  • Does a certain action hang or take too long?
• How did my app fail the Direct 3D test?
  
  • It is a C#/XAML app that only uses standard controls, and should not have any DirectX requirements.
  • It was submitted as a neutral app, and accepted all DirectX levels
• Why did none of these failures occur when running the WACK (Windows App Certification Kit) on my own computer?

All of these are pretty basic questions that a developer would need to know to reliably be able to fix the issues that the store is finding. Staying positive though, I took this as an opportunity to really spend a few days doing nothing but polish work on the app. I went through every section and feature, cleaning up and speeding up anything I wasn’t 100% happy with. In the end, I felt confident that the app was truly store-ready, and that my app was better for the whole ordeal. That is, until it failed again for the same reasons! (minus the web site issue)

Bringing In Some Help

After trying a third time, releasing only for x86 (thinking there could be an issue of ARM devices, which developers weren’t able to test at that point), I couldn’t think of anything else to do with the vague results I was getting.  So, I reached out to Bob, a developer evangelist that I met at the Hackathon at the Museum of Science.  He asked me to send him my appx package, so that he and others could look at it themselves. and to post my questions in the support forums.  I did both, and got many suggestions from various people, but everyone was basically just guessing, and telling me to try something and re-submit.  Since access to the store was open by now though, each submission took longer and longer to come back.

One person from the forums who worked at Microsoft actually had his own test store, and asked me to send him my app to run through certification there.  I was excited to have someone looking closely at my app, who could give me a more detailed explanation of what was causing the failure, but no such luck.  The app passed in his store, giving me no insight into the failures in the real store.



I was then given a link to Microsoft's support site (Windows store developers get two free questions that will be answered by someone at Microsoft, so definitely check it out) and while the person I originally talked to couldn't give me any answers, he escalated the issue to someone on the store team.  It took a few days, but he did get back to me through email, and wrote the following:

Hello Jeffrey, 

I've reviewed this and found out what the concern was. I need you to resubmit the app to make sure that the issue has been resolved. I will be following the process of this submission so that I can verify at which stage the failure should pass. If it does not I will be following up with you to resolve. 

Thank you,
-John

Without any indication as to what the issue actually was, he seems to be saying that it was something on their end, so I resubmitted the app, like he said.

After 6 days of staring at this…



…it fails with the same three errors.

Making Progress

I eventually get to talk to John on the phone.  He tells me about the little information he can get from others on the store team, and even from some of the server logs used in testing.  He says that it is failing on an ARM machine and a low-powered test computer, due to it crashing, and that because it crashes, the other two tests fail as well (which doesn't make sense to me).  He still can't explain why the app appears to pass all tests in the technical compliance phase, but then fails in the content compliance phase though.  His best advice to me is to try and test on the slowest computer possible, see if I can find any issues, and if that doesn’t work, just re-submit so that he can talk to certain people if it fails again.



With the amount of time it took last time to simply fail again, I wanted to try one more time to find out what the issue was before re-submitting.  I was already testing on a slow netbook, as well as an even slower VM, but apparently that wasn't enough.  I then went to my VM settings and crippled it even more.  I lowered to RAM to 1GB, slowed the CPU to roughly 1Ghz, and turned off 2D and 3D acceleration.  Running the app didn't turn up anything new though, as it launched with no issues, and quickly as well.  Thinking that there wasn't anything else I could do, I created a new app package to submit, and ran the WACK (Windows Application Certification Kit) one last time when… BINGO!  It finally failed a test on my computer!

Closer, but Not Quite There

Memorylage failed the WACK for taking 2.46 seconds to suspend, with a limit of only 2.  It also stated that the app used 1.5MB of IO during suspend, which was important, since my suspend code only saves a single string (the standard code from Microsoft for saving navigation history), meaning that there must have been contention trying to suspend while loading a folder.  With this information, I had a fix in under a half hour, but for some reason the store kept hiding it from me in the error reports, keeping me from finding it.

Since I was also had an un-easy feeling that the Microsoft Ad SDK I was using was causing the Direct 3D failure, I took that out as well and re-submitted.  Both the performance and Direct 3D tests did indeed pass this time, but it was still crashing on startup.

While I was heartened to see that 2 of the 3 failures were cleared up, that crashing failure (which I had NEVER seen myself) was starting to feel like it would keep me from ever getting into the store.  After a few days of moping around, I read an article that mentioned that the app store had over 3,600 apps, and Windows 8 was only 3 weeks away from general release.  This pushed to me take one final, and thorough look through ALL of my startup code.

I literally yelled (scaring my wife) when I found the issue staring at me in the very first place I looked.

In the constructor of App.xaml.cs was a call to CurrentAppSimulator.LicenseInformation that was left over from some initial test code I had written (all license code in use was using the correct object).  Since the constructor normally never has any real code in it, and is usually collapsed, I had never looked before to see that it was left.  Remembering that I had read about how calling that object will cause your app to crash when released in the store (but not when using it outside of the store) I knew that this had to be what was causing it to crash in the store every time.  After a little more checking to be sure there wasn’t anything else, I re-submitted to the store, and waited 10 days until… it failed again!

[Edit: I know that it was my fault for missing this line, but it would have been nice if it was caught by the WACK, or at least if the error report told me I missed it.  It was actually listed in the top 5 reasons for failures in the store, so I am not the only one.  Regardless, even after fixing it, it still failed.]

Another Valiant Attempt at Escalation

Ever since my third failure, I had been told by about 3-4 different people that my issue was being escalated internally, and that it should be resolved soon.  This never actually yielded any results for me, but after emailing with Allan yet again (who has done everything he could to try and help), said that it would be escalated to his boss’ boss’ boss’ boss (a corporate VP).  Needless to say, my hopes didn’t get too high.

Conveniently though, I got the failure notice only a few hours before going to a live recording of Dot Net Rocks (probably the most popular podcast on developing with Microsoft’s technologies) on the 17th.  I got there early, and got to meet the hosts, Carl Franklin and Richard Campbell, and even show them my application (I brought the Samsung Slate I had won) and told them about what I had gone through so far.  The guys were awesome, and really sympathetic about my issues, and actually quite mad about how Microsoft was handling things with the store. They said they would try and find someone that could help me out (their international tour was sponsored by Microsoft, and they constantly interview people from the company, so they have connections), but as this was just another promise of escalation, I kept my hopes low.

I emailed them the next day though, just to follow up, and thank them for taking the time to talk to me.  Richard emailed me back the next day saying that he yelled at one of the developer evangelists in New York about what developers are going through, and that hopefully the message would make its way to Steve Sinofsky (the president of the entire Windows division).  He then said that there are thousands of apps stuck in the approval process, which is why it is so backed up.  I didn’t think one more evangelist hearing about it would change anything though (I already had three tying to help me), and thought that would be it.  A few minutes later though, I got another email from Richard saying that he thought about it again, and decided to email Sinofsky himself.

This was definitely more than just a random “escalation”, so I started to get a little more hopeful, and Sinofsky even replied the next morning, saying that they would get back to me directly.  This definitely got me hopeful, but yet again, I seemed to have been let down, as I have not heard anything more from him or anyone else about it. (In Sinofsky’s defense though, he does have a bit of an important launch in a couple days to focus on).

One Last Ongoing Hope

Allan has been keeping me up to date on what he has been doing on his end, and he just had me re-submit my app today, and forwarding me an email from someone saying that they would prioritize my app when it was back in submission.  I still don’t know what was causing the last failure, and I haven’t heard back since, so I doubt I will make it in before launch, but I could be wrong.

Either way though, nothing can the fact that the current approval process is hurting developers, and that is something that Microsoft cannot afford right now.  I have already seen some progress as I have gone through my various submissions, and I am sure that in 6-12 months, none of these issues will exist, but I don’t think Microsoft can afford to wait that long.

The Short Version

What I Did to Get Ready For Submission

• I started working on Memorylage back in December, with the Developer Preview.  I have been programming for Windows for 17 years, with C# for the last 6, so I wasn’t starting from scratch
• I read through all of the their design and coding documentation, trying to really create something that showed off Windows 8’s features, and its design style
• I re-wrote the app for the Consumer Preview when it was released, since a lot had changed, and the new version of Visual Studio wouldn’t even open up files from the Developer Preview anyways
• I attended a 3-day App Excellence lab at Microsoft’s Waltham MA office, where Memorylage won the App X challenge at the end of it
• I also setup a meeting with a Microsoft Field Engineer to review my app, so that I could get early access to the store.  This process was extremely detailed, with a 60+ point checklist of criteria that my app needed to meet to get it.  I had three things that I needed to fix when I first met him, but I passed the second time
• I continued working on the app without actually submitting yet, because I wanted to make sure that it was fully polished and feature complete
• I signed up for Microsoft’s 30-To-Launch program, which helps developers go through all of the steps necessary to get into the store.  I already had gone through everything that they showed, but I figured it couldn’t hurt anyways, in case there was something new
• I went to a Microsoft hosted Hackathon at the Museum of Science in Boston, where Memorylage again won the App X challenge at the end.  This time, I netted a $1,200 Samsung Series 7 Slate with dock, a $500 gift card and an XBox game
• I met several Microsoft people (and others) there that were encouraging me to submit my app to the store immediately, since they thought it was ready, and was something they wanted in the store
• After another two weeks of polishing, I finally submitted my app on August 29th (almost a full two months before the release of Windows 8)

What Happened After Submission

• The app failed for 4 reasons, 3 of which gave me no real idea of what caused the failure.  The failures were for stopping responding (when did this happen?), performance (performance of what?), Direct 3D (I don’t use Direct 3D) and having an incomplete web site (I had it finished a day later)
• I went through all of my code, cleaning up performance anywhere I could, and tried to fix anything that I wasn’t 100% happy with
• I submitted again, and failed again for the same 3 reasons (minus the web site issue)
• I emailed Bob, a developer evangelist who I met at the Boston Hackathon, who said that he would escalate the issue internally
• I re-submitted again, but only for x86, thinking it may be an issue with ARM (since no developers had access to ARM to test with), but it still failed for the same reasons
• I posted multiple thread in the support forums, asking for help with what I thought was an issue with the startup time (the performance failure) and the Direct 3D failure
• Matt, who works at Microsoft and replied to the app startup post, asked me to send him my app, so he could test it in his own personal store.  It passed all tests for him though, so I didn’t find out anything new
• I went to Microsoft’s support site, and used one of my two free help requests to talk to someone directly through chat.  They couldn’t help me, but escalated me to someone on the store team, who would get back to me via email
• A few days later, I got an email from John, the person my support ticket was escalated to, saying that he thinks everything should be all set now, and to just re-submit.  He never said what he thought was wrong in the first place
• I resubmit and fail again for the same reasons.  This time, it took 6 days to fail, since everyone had access to the store now.  Each submission was now becoming a lot more costly as the release of Windows 8 was starting to get closer
• John tells me that the failure is because I submitted a neutral app, which will be tested against ARM, but that can’t be it, because I had already tested as x86 only (the third submission).  He also said that the Direct 3D failure was because I didn’t have a check against the feature level, and not because I used any Direct 3D features.  This was also incorrect, as the documentation clearly states that you only need to check the feature level if using Direct X 10 or up
• I get to talk to John on the phone, and he tells me that they couldn’t find out much information about my failures (even after pulling server logs), except that the app was apparently crashing during startup on a low-powered machine
• I cripple my VM, so that it is SUPER slow, and run the WACK (Windows Application Certification Kit), which finally causes it to fail.  The failure is due to taking too long to suspend, and the information it gives allows me to fix it in under a half hour.  Why the store hides this information, and just gives a generic “Performance” error is beyond me
• I also remove the Microsoft Ad SDK, since I have a feeling that it might be causing the Direct 3D failure
• I re-submit, and wait another 5 days for it to fail again.  The performance and Direct 3D issues were gone though, so it was just the crashing failure that was left
• I find that my code has a leftover call to CurrentAppSimulator.LicenseInformation from initial development of the licensing code, and I know that this will cause the app to fail certification, but has no effect when running it on my own (making it very hard to catch) [Edit: I know that it was my fault for missing this line, but it would have been nice if it was caught by the WACK, or if the error report at least told me that I missed it.  It was actually listed in the top 5 reasons for failures in the store, so I am not the only one.  Regardless, even after fixing it, it still failed.]
• I re-submit, and am hopeful that this will have fixed all of the issues, but after 10 days of waiting, it fails again with the same crashing issue
• Allan (another developer evangelist that had been helping me a lot) says that he has escalated my issue to his boss’ boss’ boss’ boss (a corporate VP), and will still try to get it through before launch (this is now the 17th of October, 9 days until launch)
• I meet Carl Franklin and Richard Campbell from Dot Net Rocks, at a live recording in Boston, who I got to show my app to, and tell about my issues.  They say they will try to get someone to help, and initially just talk to some developer evangelist that they meet in New York about it, but Richard then decides to email Steven Sinofsky himself (head of the entire Windows Division) about the issue
• Also, while at the Dot Net Rocks event, I meet Chris, who ran the App Excellence lab I went to, and he let me test Memorylage on his ARM tablet, and it worked perfectly, so the issues were not ARM based
• Sinofsky actually responds the next day, saying that they will get back to me directly (I never hear back though)
• I attend a webcast from Microsoft about common pitfalls of submitting to the store.  The issue of calling the CurrentAppSimulator object is in the top 5 causes of failure, but thry still won’t catch that in any of the tests, or even tell you that was a cause for failure.  The developer is left to figure it out on their own
• John asks me to send him my app again, so he can look into it again (I haven’t heard back from him yet)
• Allan tells me that I am on the docket for some critical store meeting, so they are working on it
• Allan forwards me an email, asking me to re-submit my app, and they will prioritize the certification process
• I am still waiting to hear back on that (it is now just past midnight on the 25th, with just under 24 hours until launch), but that was only yesterday, so I am actually confident I will hear back soon
• So far, I have sent/received 131 emails back and forth with various people within Microsoft
• I have given my app, and even my source code, to multiple people within Microsoft.  To date, not a single one of them has been able to cause it to crash, or fail a single test.  Every issue that I have figured out, I found on my own, but the store still fails it.  How do you debug that?

Conclusion

What Went Wrong

There are many reasons why my app is still not in the store, and many things that could be improved.  This is just a quick list of the ones I could think of.  Hopefully, people from the store team will see this, and work on improving things.  I have already noticed some improvement between the various submissions, but there is still a long way to go.

• Failure reports need MORE INFORMATION.  Imagine hitting compile on a large program, and instead of the compiler saying that you missed a semi-colon, and on which line, it waits a week, and then just says that you have a syntax error.  That is what the certification process feels like
• Do not hide information from the WACK.  If an app fails the suspend test, give the developer that information.  It already creates a detailed report, so just forward that along.
• If a test fails, especially a performance related one, or a crashing issue, let the developer know what architecture it failed on.  As developers currently can’t test on ARM, letting them know that the app works great on x86 but fails on ARM would be really helpful.  Same with if it passes on a normal machine, but fails on a very low-powered one
• I still have no idea why the Ad SDK causes the app to fail the Direct 3D test.  Everyone that I talked to says it shouldn’t, as it only uses a web viewer.  If it does have an issue though, then developers should be notified, so they aren’t left guessing
• Check for use of CurrentAppSimulator in the WACK.  This was one of the top 5 reasons that apps failed validation, so it is a common mistake.  Having the WACK check for it allows developers to find it before they even submit.  At the very least though, tell the developer that it is the cause if it fails.  None of my error reports gave any hints about it
• When an app fails in the Content Validation phase, where there is an actual person looking at it, they should spend a few minutes giving as detailed a response as possible.  Every response I got was a single sentence that told me nothing.  It may take them a little longer per submission, but if it keeps developers from re-submitting 7 times, it will speed things up overall, and keep developers happier

What I Lost Because of This

Even if my app does get into the store before launch, or very close to it, there are already plenty of things that I have lost, which can’t really be recovered.  Hopefully others have been luckier than me.

• Almost 2 months of marketing for my app, with very little competition.  When I first submitted my app, there were only 500 apps.  Now there are over 7,000
• Almost 2 months of development time.  I really couldn’t focus on adding new features when the current ones won’t pass.  When dealing with a new market place like this, development time is precious, and 2 months is a lot
• Another Samsung Series 7 Slate.  The 30-To-Launch program is giving out a free slate to everyone who registered and got their app in the store before launch.  Even though I submitted 2 months early, with a completed and polished app, I lose out on $1,200 worth of hardware
• Extra marketing from winning the Boston Hackathon.  Microsoft is offering to highlight Hackathon winners that have made it into the store, but the deadline in next Friday, and even if I make that, I have already lost the time over the past 2 months
• Downloads and reviews.  If I had been in the store all this time, I would already have built up enough of a reputation to help bring my app higher in the search results, instead of starting from scratch.

In Conclusion

This post went on waaay longer than I planned so I will be brief.

I still think that Windows 8 is a great opportunity for developers, but as it stands, they are in for a world of hurt in trying to get through that last hurdle.  As a long-time Windows developer, I really hope that changes soon.

Also, I want to say that while the overall process may have been negative, every person that I talked to personally was always helpful, and willing to go out of their way to help me (especially Allan), and I don't want to put them in a negative light.  It is just that Microsoft is a large company, and despite their best efforts, things still turned out the way they have.