Skip to main content

Wow reclist?  Thanks guys!!!  :-)  Lots of good input from IT folks coming in...

As someone with nearly 30 years in IT and having worked on federal IT implementations for most of that time, I have a few observations on the "debacle" that occurred during the rollout of  Follow me below the orange Mobius for some discussion...

There are a number of alleged flaws to the design or architecture of the system.  In addition, there are some assertions about how the companies involved came to be selected.

1.  The vendor

Apparently it is being asserted in right-wing blogs that this was a sole-source procurement.  

Anyone who knows federal IT can tell you that sole-sourcing new development is almost impossible.  Even sole-sourcing operations and maintenance (O&M) of existing systems is rare.  The Federal Acquisition Regulations (FAR) make that very difficult.  

It is far more likely that this contract went to the lowest bidder.  Despite the evolution of "Best Value" procurements, far too often federal contracting offices interpret it as lowest cost, either because the RFP didn't correctly weight technical factors over cost or didn't probe deeply enough into delivery capability.  And again, given the FAR, and the type of procurement, the contracting folks may have had limited flexibility in that regard.

I stand corrected.  This WAS a sole-source.  But in hindsight, given how long and expensive a procurement can be, this may have seemed like a smart decision at the time.

2.  The architecture

Here I think there may be legitimate concerns.  This appears to have been, more or less, an attempt at a service-oriented architecture (SOA).  Given the way this thing is supposed to provide real-time validations of inputs via its "data hub" connecting to remote systems, there could be many points where the systems exposing the data validation (presumably via web services) could have been overwhelmed.  What is apparent is that the crucial service choreographer component did not gracefully handle the failure of a service to respond.

This could have been a function of the process design, which in turn came from the requirements provided by the government.  Ideally, you would want the system to TRY and validate the input, but if it fails, simply go ahead to the next step but flag the record for subsequent review (what we call a "human task").  This is especially important for synchronous calls where processing on a thread is halted until a response is received.  But if that wasn't specified in the requirements, or especially if it was explicitly ruled out, then such graceful handling of an error condition would not have been implemented.  If the people in the government insisted on a completely automated data validation process, they exposed themselves to this risk.  Whether the vendor understood this and raised it as a risk, is something we are unlikely to discover.

3.  Website or app?

As far as turning it into an app, it is not so easy.  Apps suck for significant data entry because phones are not designed for typing.  Furthermore, there are limits to what "responsive design" can accomplish when you have to enter a lot of data, plain and simple: screens that work great on PCs or Macs will be virtually unusable on small form factor devices.  

Yes, the youth of today are more likely to use phones for internet access, and this is particularly true of less affluent youth.  But the fact remains, some things are just not good on mobile devices, and it really is a limitation of the devices rather than the applications.  There is such a thing as fit for purpose.

4.  Government fails again

I have read a lot about how this just shows how incompetent the government is in general, and with specific reference to IT in particular.  I think this is horseshit, frankly.  The primary difference between government IT failures and corporate IT failures is that the former tend to be on the front page of the Washington Post whereas the latter do not.

For example, how many of you endured the debacle that was Virgin America's switchover to the Sabre reservation system?  This is a system that most airlines use and it should have been smooth and easy, but instead features of their website didn't work for days or even weeks.  Mostly this surfaced in the tech press because Virgin is practically the house airline for the tech community (full disclosure: I absolutely LOVE Virgin America but a fail is a fail).

Here is a list of 10 Famous ERP Fails that again most people don't know about.  (The last one in the list is kind of a gag.)  But again, this stuff happens all the time in complex implementations.  

I've seen a lot of people write that "we should have had Google or Amazon engineers build this."  Well, look at the outages of the Amazon and Microsoft clouds.  These are companies that have spent billions of dollars on infrastructure, and it is a core competency for them.  And yet, stupid things happen and boom, down it goes, for hours, sometimes many hours.  And Facebook has had it share of coding fails, where they expose such and such or whatever.  These companies tend to fix their problems quickly, but the problems still occur.

My personal favorite is the Romney campaign's abysmal tech failure.  You could write a textbook on that one.

5.  In Conclusion

I am sure there are many people involved in Healthcare.Gov whose individual decisions were quite rational at the time given what they knew or thought they knew, but who with the benefit of hindsight would make a different choice.  That being said, the unexpected always happens, and the measure of individuals, companies and governments is how they handle it.  If they get this fixed in a couple weeks, six months from now few will remember it.  But it is NOT another example of a government failure.  Failure is part of technology.  That's why it is wise to always be at least a bit skeptical about betting the farm on it.


Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags


More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  Tip Jar (130+ / 0-)

    -9.00, -5.85
    Quite an experience to live in fear, isn't it? That's what it is to be a slave.

    by Wintermute on Mon Oct 14, 2013 at 01:08:10 PM PDT

  •  heh (15+ / 0-)

    phones may not be designed for typing, but that doesn't stop half the teens and twenty-somethings from typing 40 words a minute on them.

    The cold passion for truth hunts in no pack. -Robinson Jeffers

    by Laurence Lewis on Mon Oct 14, 2013 at 01:12:03 PM PDT

  •  Most of that sounds pretty reasonable. (12+ / 0-)

    I have submitted, in a comment or two, that a good chunk of the 'rollout' problem was that the implementation deadline set was seemingly set for political reasons, rather than because a timeline analysis of the actual tasks involved suggested that they would be done with time to spare by the chosen time.  (Ie, the people in charge, who are not IT people, set the deadlines based on their own desires, not on the actual amount of time needed to first code, than test the product fully.)

    Your response to that suggestion?

  •  Can they just copy what California did? (0+ / 0-)
    •  California has its own... (3+ / 0-)
      Recommended by:
      ferg, Boston to Salem, sethtriggs

      as all the other states have their own. They all have individual business models much less diverse technologies...

      •  Ca site works for 38 million people, it could (3+ / 0-)

        be scaled up for 314 million. The fact that California can do it, means it probably can be done. They should ask California for help, the developers should swallow their pride and do that.

        •  But you can't just take the California site and (0+ / 0-)

          graft it onto the Federal site. You'd have to totally replace one with the other, which would mean taking things off like for quite a while. There may well be a much quicker fix, even if it is just a workaround. I wonder about limiting the number of phones that can connect at a time, to speed up access for traditional computers. Which in turn would encourage many of the smartphone addicts to use a computer, even if they have to do it at a friends house, or a library.

          Just another underemployed IT professional computer geek.

          by RhodeIslandAspie on Mon Oct 14, 2013 at 03:44:56 PM PDT

          [ Parent ]

      •  Not all the states (1+ / 0-)
        Recommended by:

        Remember, dozens of red states refused the federal money not only to expand Medicaid, but also the federal money that was offered to set up their own exchanges.

        So ironically, the red "get the federal gubmint out of my Medicare" states opted to let the federal gubmint run their exchanges.

        Also why got more hits than it was designed for....

        "Mediocrity cannot know excellence." -- Sherlock Holmes

        by La Gitane on Mon Oct 14, 2013 at 05:20:20 PM PDT

        [ Parent ]

  •  Regarding #1 (2+ / 0-)
    Recommended by:
    lgmcp, WisVoter

    It appears they modded an existing contract with CGI Federal.

    To believe that markets determine value is to believe that milk comes from plastic bottles. Bromley (1985)

    by sneakers563 on Mon Oct 14, 2013 at 01:40:12 PM PDT

  •  I just wish it could be fixed quickly. This (7+ / 0-)

    is the last thing the ACA needed and gives the Repugs fodder.  I am a single payer advocate but the ACA is what we have now so I will support it--for now.  Too many are left with no coverage and here is WI with Walker not taking the Expanded Medicare--I fear the health of the state is doing down down down.  

  •  Not so competitive bidding? (8+ / 0-)
    "The contract for wasn’t a fully competitive process. In fact, there was no competition for it at all. To get the work done, the Department of Health and Human Services used an existing contract it already had with CGI Federal to get the work done. The contract, as they say, was 'greased.'

    “Texas is a so-called red state, but you’ve got 10 million Democrats here in Texas. And …, there are a whole lot of people here in Texas who need us, and who need us to fight for them.” President Obama

    by Catte Nappe on Mon Oct 14, 2013 at 01:42:28 PM PDT

  •  This front-end dev at a tech startup agrees (20+ / 0-)

    My specialty is front-end but I'm part of a small team of engineers who span from back-end to middle-layer to me. :-)

    I agree with your assessment. Tech fails, and luckily we have ways to dissect the problems and correct them. Perhaps the team underestimated demand, but nonetheless we need to trust that they are working to fix it.

    Also, remember that government websites need to accommodate a wider range of web browsers, perhaps all the way down to Internet Explorer 6. Smaller stsrtups with demographic data have the luxury of choosing how far back they want to support legacy browsers, whereas government agencies do not. It's the online equivalent of public transit.

    As such, those older browsers have tons of bugs in themselves which makes it increasingly difficult for modern websites to run efficiently and accurately.

    •  Agreed... (15+ / 0-)

      front-end here too....and not just legacy browsers...legacy code from the various websites that are accessed. Legacy databases etc.

      Anyone in the business understands this is complex and letting users find the flaws is the best way to find and fix them.

    •  Arggh... IE6.... The horror!!!! (7+ / 0-)

      And, that is not being sarcastic.  IE6 is the bane of my existence, if my team did not need to support it there are a lot of issues that would just go away.

      One thing that was probably missed is that in something like this test automation and deployment automation to allow for a change to be made, deployed for testing, tested, and then deployed to the outside world in a day or so along with easy rollback to the previous version just in case would probably have been valuable.  Along with green/blue deployments.  For those not in the know that is being able to make hot deployments to limited users to verify before rolling out to everyone.  Then, fixes would likely show up much faster.  That is the real difference that the best web application companies provide.  They make the same mistakes, but instead of big high risk rollouts they do small frequent low risk rollouts that can be easily backed out if something goes wrong.

      Hearing things like the problems won't be solved for weeks means that where ever the problem is, the fix cannot be deployed or verified quickly. Because usually tracking down and fixing a problem is the easy part.  Making sure you didn't break something else in the process is the hard part.

    •  IE6 is off the table (2+ / 0-)
      Recommended by:
      BlueMississippi, mattc129

      Less than a quarter percent share according to Statcounter.  Same for IE7 and less that one percent.

      Economic Left/Right: -7.38
      Social Libertarian/Authoritarian: -8.00
      Two steps to the right of Trotsky.

      by jvance on Mon Oct 14, 2013 at 05:05:24 PM PDT

      [ Parent ]

    •  I've done both legacy back-end (4+ / 0-)

      mainframes and front-end Windows and web, and I agree. Don't forget that a lot of "back-end database" is often accomplished by calling legacy COBOL programs to retrieve data from VSAM files on the "database" may not even be a database. In addition, during my experience in government IT many years ago, there WEREN'T the "off-the-shelf" solutions yet, so every department wrote their own programs, using their own standards. Coordinating that all into a new huge system must have been a nightmare.

      Some of the mainframe stuff may have been updated into true databases (IBM mainframe is usually DB2), but if it ain't broke, you usually don't fix it, because it's too expensive.  

  •  I'm an IT system analyst (13+ / 0-)

    What you said.  

    Never doubt that a small group of thoughtful, committed citizens can change the world; indeed, it's the only thing that ever has. - Margaret Mead -

    by FlamingoGrrl on Mon Oct 14, 2013 at 01:50:54 PM PDT

  •  If you already have health insurance, (33+ / 0-)

    give it a rest. There is no need to worry about the ACA website. I see so much concern-trolling about the website that it just makes me want to rant. (For the record, I am not saying that THIS diarist is a concern troll. In fact, I am specifically putting this comment in this diary because I think it is NOT, in fact, concern trolling.)

    But seriously, people, if you have insurance, lay it down. If you don't and you are worried, I sympathize and empathize, but even you have plenty of time so pick up the phone. I am lucky to live in a State that has its own website (with its own problems) but no matter, like you I have the option to do the application over the phone or by mail. Take advantage of that. It will be financially worth your time investment especially if you are in a state that has agreed to expand Medicaid.

    There are still 2 whole months before the first deadline for coverage. If you don't need insurance, don't clog up the website with your traffic and quit crying "the sky is falling", because the exact opposite is happening for people like me. (Uninsured, self-employed with history of cancer) My life is now a whole helluva lot less fraught. Problems with the website don't amount of a hill of beans when compared with my new insurance policy.

    Oh, and if you are not in a state with expanded Medicaid, get on the phone to your governor, state legislators, senators and congress people and don't stop calling until they hear you.

    •  ..2 months (6+ / 0-)
      Recommended by:
      greenbell, jorogo, Kevvboy, ybruti, ChadmanFL, kurt

      seems like a long time...but is it?  The problem I have is that there was no backup plan?  You can only try to finalize your application through the website process.  Even the navigators and ACA customer service are straddled with this "tech" glitch.

      If you know you are going to have "glitches" as every program rollout does, then why do you not have a plan for going around the website?  Why is there not a field office in my state (NC which can only use the federal exchange) were I can go and see a representative..give them my info...have them give me my options...I choose a plan...they start my plan...and at 12:01 am Jan 1st I am covered?  

      The backup plan to a failed website shouldn't be the failed website?  thats just idiotic.  and if you delay by 1 month or 2 months peoples accessibility to the health exchanges then you need to extend the cutoff date past March....

      •  You need to blame your state just as much (0+ / 0-)

        SCOTUS threw a wrench into the works by letting states refuse Medicaid expansion and smaller wrenches came from assorted non-cooperative states making it harder for navigators to do their work. You can't lay all that on the FED's door -- and even if it did lay at the Feds door, don't whine about the website, get on the phone to a navigator to someone else who can help.  

        If your state is preventing you from accessing a navigator or other aid, make some serious noise at the ballot box next time around. But don't wait, get on the phone with them now.

        •  i have talked (4+ / 0-)
          Recommended by:
          greenbell, flo58, chimene, jorogo

          to them...its not their fault.  They don't have anymore access to putting my application in then I do.  You are making assumptions about something you obviously haven't had to deal with.  I am 100% behind this working, but they gave no one the tools to allow that to happen.

          ACA representatives and navigators should have been given the ability to get me into the system that I can't currently access online...they don't.  They have to use the same access I have.  Thats idiotic.  The system as it is constituted today is that ALL of us including those who are taking our info have to use the "glitchy system"...why can't I call an ACA representative or navigator...have them walk me through the application...they enter it and I then have direct access to view the exchange and see the different plans...then choose a plan and have it effective Jan 1st.  Right now that is not applicable.....State Exchanges are getting hung for this same reason...lots of states such as Maryland are getting bottle necked at the Federal Level...Its effecting all the exchanges.....thats the part I want to fix...I don't care about the website..that will be done in due time...its the fact that they didn't have personal access way to do all this thats got my painties in a bunch...its just idiotic to have an all data verification that has to go thru 6 agencies and not have a way around the digital process......

          •  Understood (3+ / 0-)
            Recommended by:
            amsterdam, chimene, jorogo

            I just wanna get the party started. I (literally) can't wait to see Obamacare implemented. I truly hope the site gets fixed and -- everyone who wants to -- gets signed up by December 15. I have a boatload riding on it.

            •  thats the other (4+ / 0-)
              Recommended by:
              flo58, TX Freethinker, chimene, tofumagoo

              thing that has my panties in a bunch.  I have a lot of co-workers and friends who do not currently have health insurance...and I want more than anything to brag about how they can now afford it and mean really afford it.  They would sign up in a heart beat..that I am for certain...the prices for us getting subsidies is really they will def. sign up. But at this point I am glad I didn't already brag about the site before it was implemented...I WANT TO SELL THIS AS LOUD AS I CAN TO ALL I KNOW..but right now I have to keep it a low profile until they get this worked out.  Hopefully sooner rather than later......

    •  That's ridiculous. (4+ / 0-)
      Recommended by:
      greenbell, jorogo, chimene, ChadmanFL

      There is a political component to ACA if you haven't noticed and the exchanges are the public face of it.  This was everyone's first impression and it was abysmal politics. The thinking is simple: if you fucked this up, what else did you fuck up. And you have to wonder.

      This made a very bad first impression and I will bet you dollars to donuts they don't have this fixed in all states by the first deadline for coverage.

      This affects me personally as I have been unable to apply.

      If you hate government, don't run for office in that government.

      by Bensdad on Mon Oct 14, 2013 at 02:47:55 PM PDT

      [ Parent ]

    •  2 months (4+ / 0-)
      Recommended by:
      greenbell, jorogo, ChadmanFL, kurt

      A half month already went by with me not able to verify my identity. That means I'm stopped in my tracks, unable to proceed.

      Having two whole months is small consolation if there's no guarantee that they're going to get it working.

    •  I do already have health insurance, (5+ / 0-)
      Recommended by:
      Dvalkure, ChadmanFL, chimene, tofumagoo, Ian S

      and it's bleeding my modest, fixed-income budget dry. I literally can't afford to "give it a rest". The website fail is a real issue affecting my ability to plan my future.

      I tried the phone - last Monday, with a nice man at the helpline who went over the same questions I submitted online 3 times, only to have that fail. The phone submission failed, too. Another helpline person found no registration for me.

      I'm trying paper now, too. They can get a paper form to me by....October 31, the same day my current insurance has a deadline for me to lock in a slightly higher rate for a year to avoid anticipated large increases at my anniversary date. Why there are not forms in every post office and library just like tax forms, I don't know.

      You'll be glad to know I have made calls to my government reps, but I'm still stuck without options and my own deadlines looming. I'm so sorry all that is upsetting to you.

      "All war is stupid" - JFK

      by jorogo on Mon Oct 14, 2013 at 03:41:37 PM PDT

      [ Parent ]

      •  Apologies... (6+ / 0-)

        I was talking about people who have insurance through jobs or medicare or who are otherwise unqualified for the ACA. I can totally appreciate your trouble and hope they can get things fixed ASAP. I wish I could help you with the issues.

        My concern is different than yours. I just really don't want the implementation of the law to be delayed. That is what I fear that many people want, who are focusing on website issues over the delivery of care. Why, you ask? Well, because depending on how that delay happens, I worry it could cost me (and people like me) my/our lives. I am a self-employed, uninsurable cancer survivor who has just received a new cancer diagnosis. I need surgery, I need months of chemo, I need radiation, not to mention massive numbers of tests and who knows what. And I can't get any of that at the emergency room.

        Call me selfish, but I want treatment -- and to get insurance that will help me pay for it. Along with that, I want the discussion to focus on the delivery of care as much as the problems with the website. When people who are not affected by the law can do nothing but complain about technical issues, well, yes, to quote you, "all that is upsetting to [me]." And I am not about to apologize for that upset.

        Good luck getting a good and cost-effective policy on the exchange.

        •  And best wishes for you, too. (0+ / 0-)

          Our common ground here really shows that there is dire need for reform, so that health insurance actually fulfills what it is intended to do.

          I've had 2 back surgeries, one on my current policy, and every month I've been holding my breath anticipating a cancellation notice. I really wanted single-payer, or at least a public option, but I'm withholding criticism on that basis because I appreciate that the fear of cancellation is being lifted from me. That alone is worth going forward with the ACA.

          "All war is stupid" - JFK

          by jorogo on Tue Oct 15, 2013 at 02:51:43 PM PDT

          [ Parent ]

      • (3+ / 0-)
        Recommended by:
        flo58, amsterdam, jorogo

        Have you reached out to any local navigators or assisters? Yes, for goodness sake, the website should work -- but there are alternatives. Some agencies in our area are helping people on a walk-in basis.

        •  I selected the "find local help" button (0+ / 0-)

          on the website. It listed the nearest navigator help as a County Health and Human Services office 2 counties away. I called anyhow, and the lady there said it was unfortunate that the website listed them without consulting them because they do not provide that service.

          Just another in a litany of mistakes for this botched roll-out.

          "All war is stupid" - JFK

          by jorogo on Tue Oct 15, 2013 at 02:57:08 PM PDT

          [ Parent ]

      •  Why can't they email the form to you? (1+ / 0-)
        Recommended by:
        •  why can't you print it off the internet? (1+ / 0-)
          Recommended by:

          "real" work : a job where you wash your hands BEFORE you use the bathroom...

          by chimene on Mon Oct 14, 2013 at 08:59:50 PM PDT

          [ Parent ]

          •  I could, (0+ / 0-)

            but it's not there to be downloaded, at least not at the stage of application that I've been able to advance to.

            I even called my U.S. Senator's office (the smart one - Baldwin, not the dumb one - Johnson), and they don't have access to the form.

            It's almost surreal, isn't it?

            "All war is stupid" - JFK

            by jorogo on Tue Oct 15, 2013 at 03:04:07 PM PDT

            [ Parent ]

        •  "We don't have access to email." (0+ / 0-)

          That was not a typo. I asked for that to be repeated to me on the phone. They do not have access to email at the helpline or at the advanced resolution center.

          "All war is stupid" - JFK

          by jorogo on Tue Oct 15, 2013 at 02:59:31 PM PDT

          [ Parent ]

  •  Thank you!!! (11+ / 0-)

    I like to think that Kossacks prefer hearing from actual
    "experts in their field" prior to making a judgement about how big a "screw up" something was.

    I know I really appreciate getting your take on this.

    Lisa :)

    All Kossacks are my allies, but if you can't express your thoughts in a civil and kind manner, I won't be engaging in a conversation with you.

    by Boston to Salem on Mon Oct 14, 2013 at 02:18:00 PM PDT

  •  A lot easier to build bad software than good (5+ / 0-)

    and from what I'm hearing this one isn't all THAT bad.  

    But I didn't go clog it up because a) I'm fully insured and b) I was more interested in testing my state's new Exchange page.  

    In terms of the prices I saw our state page quoting, it looked like the gold plans were a LITTLE bit cheaper than what my employer and I together pay for my gold plan.  In other words, with no employer to pick up a hefty share, even favorable large-group pricing still looks not so wildly affordable.

    "The extinction of the human race will come from its inability to EMOTIONALLY comprehend the exponential function." -- Edward Teller

    by lgmcp on Mon Oct 14, 2013 at 02:20:13 PM PDT

  •  Thanks for this...I wonder what their baseline (5+ / 0-)
    Recommended by:
    closerange, sethtriggs, lgmcp, blueoregon, kurt

    expectations were for the rollout (i.e., how many Americans were eligible for using the site instead of their state's; out of this figure, how many were anticipated to sign up in the first week)?  I'd be willing to bet that they lowballed it, figuring that between those who did not have the time, would procrastinate, were unaware of their eligibility, were unaware of their state's programs, etc., that they would have numbers consistent with the capabilities of the system's design.  Instead, they to what appear to have been a deluge, swamping the system.  

    If IT holds to other business models, this is a very good thing--the designers will amp up the ability of the system to handle the flow and this is an indicator that more eligible  individuals were interested in the product than was anticipated.  

    So long as those who could not sign up yet or those reading about the problems are not discouraged, the sign up rate has the potential to far exceed expectations even before the big drive to reach out to eligible individuals who need to be educated on their options has kicked in.

    To be free and just depends on us. Victor Hugo.

    by dizzydean on Mon Oct 14, 2013 at 02:26:58 PM PDT

  •  KISS, Keep It Simple Stupid! (5+ / 0-)

    It sounds to me like they had a lot of money thrown at them and they decided to put in every bell and whistle they could get their hands on.

    Why was real time confirmation of data needed?  It wasn't! Trying to grab data from multiple databases from multiple government agencies, and doing it in real time, is as stupid as the Tea Baggers trying to defund Obamacare.

    I absolutely hate midnight batch jobs, and have converted many to real time, but when you don't need to do things in real time, and you're doing a project with such public exposure, it would have made far more sense to process applications in the background.

    Let the customers browse the plans.  Let them fill out and submit a standard form with the information needed. Processes continuously in the background and email the client, or snail mail, or even auto telephone, when their application is processed.

    Frankly, this isn't rocket science and processing 7 million applications could be done on a half a dozen Dell PC's in a few days.  

    Instead of keeping it as simple as a Google front page, they tried to make it some fancy retail outlet.  The company should be fined and barred from any further government contracts.

    •  and thats the funny (1+ / 0-)
      Recommended by:

      part about this site.  I haven't actually been able to get to where my application gets processed through the different agencies.  They kill it from the signup.  I am beginning to think that it is intentional so that you don't spend 30 minutes filling out the forms only to have it hose up when it tries to get processed through these agencies.  

      Especially if its real time data...hell if you misspell an address or zip code or put a period in the wrong place one of the 6 different agencies validating the info can send back a fail...and your kicked out of the system.... Doesn't really make any since on any level to have real time qoutes and verification if you have to go through 6 agencies plus state agencies....and to not have a back up plan..if this website fails...just makes this mind boggling..

    •  Max Baucus ensured (5+ / 0-)
      Recommended by:
      greenbell, ybruti, mrsgoo, pollwatcher, kurt

      ...that KISS would not happen.

      Keeping it simple would mean single-payer, removing the age restriction on Medicare, and upping the capacity of existing functioning software.

      50 states, 210 media market, 435 Congressional Districts, 3080 counties, 192,480 precincts

      by TarheelDem on Mon Oct 14, 2013 at 03:47:50 PM PDT

      [ Parent ]

  •  That's why Star Wars was an obvious no-no; never (8+ / 0-)

    pin the future of the earth on software working perfectly the frst time.

    Is it true? Is it kind? Is it necessary? . . . and respect the dignity of every human being.

    by Wee Mama on Mon Oct 14, 2013 at 02:40:30 PM PDT

  •  Ridiculous. (3+ / 0-)
    Recommended by:
    greenbell, Dream It Real, shrike

    1.  There wasn't just one site, there were many and that was part of the problem.
    2. Sibelius wasn't riding herd on this. If she had been she would know these sites weren't up and running. I am in a blue blue blue state and my state's site is still not up. She cannot personally fix it, but she can start screaming. And there is no excuse for not to be running even if the states were individually fucking it up.
    3. Still can''t apply via my state's site. We are so blue people fly here to get married just to see all of that blue.
    4. All of this was foreseeable and preventable.
    5. The President announced the rollout...and then fizzle. This was the public face of Obamacare and you only get one chance to make a first impression. Millions may have been turned away or turned off.
    6. It couldn't have come at a worse time.
    7. Did anyone contact those liberals in Silicon Valley for help? Don't think so.
    8. DO THE FUCKING APP, TOO. It doesn't have to be either or.

    What's to be done? Nothing. You can't fix a fuck up this bad.

    If you hate government, don't run for office in that government.

    by Bensdad on Mon Oct 14, 2013 at 02:43:36 PM PDT

    •  So everyone should just quit (2+ / 0-)
      Recommended by:
      Sherri in TX, amsterdam

      And declare it unfixable?

      I hear gardening is a nice hobby.

      by SeanF on Mon Oct 14, 2013 at 04:19:59 PM PDT

      [ Parent ]

    •  WA state? (5+ / 0-)
      Recommended by:
      greenbell, Bensdad, chimene, tofumagoo, kurt

      I'm in WA, and my application is dead in its tracks.  I discovered that the only way to get through on the phone help line is to call precisely at 7:30am.  Each time I did that, I would get someone who seemed very helpful and knowledgeable, but each time they would tell me they had fixed my issue through the back door, but it would take "24 hours"  or "48 hours" for the system to recognize it, and I should call back if I didn't see the change by then.  Each time, there would be no change.  The last time I had one of these interactions, I actually ended up being pushed further back in the stages of the application process, and now I am stuck there.

      From doing my own research, I am pretty certain that once everything works, I will be able to get a pretty good deal on an insurance plan, given the subsidy I qualify for.  However, I am rather tech savvy, and persistent, and also I WANT to believe the ACA is a good thing.  For someone who does not have all of those qualities, the experience I've had would probably lead them to think that, not only does the tech not work, but the plans being offered are ridiculously expensive.

      •  Similar to my situtation in MN (2+ / 0-)
        Recommended by:
        Bensdad, kurt

        I've been stuck at the same point for 10 days.  The Help Desk has no ability to look at my account or data.  They forward the ticket.  No one works on the ticket.  No one accessed my help ticket from the day it was entered by the Help Desk through today.  They can tell me no one is doing anything about my problem and that no one has even looked at my problem but they can't do anything but tell me no one is doing anything.  

        There is also an e-mail address you can write.  That took 8 days to respond with a form telling me they would reply in a week.  

    •  I just don't buy it (2+ / 0-)
      Recommended by:
      madronagal, amsterdam

      every computer program this complicated has problems, every one. Maybe it shouldn't be this way but it is and it does not signify anything other then a complicated roll out.

      I see the bigger problem is people concern trolling..."oh if the roll out is not perfect then..."  what?

      Really, grow up. You know full well that nothing will go into effect until the new year so you can wait a few weeks and absolutely nothing will be changed.

      Stop screaming that the sky is falling.

      •  You grow up. (2+ / 0-)
        Recommended by:
        OrganicChemist, Ian S

        It isn't "concern trolling" if people, like me, can't compare health insurance policies and then sign up and budget for next year.

        I can only imagine what those who were predisposed NOT to like Obamacare are thinking.

        If you hate government, don't run for office in that government.

        by Bensdad on Mon Oct 14, 2013 at 05:05:56 PM PDT

        [ Parent ]

  • has been a work in progress... (11+ / 0-)

    ...for years.

    It gets better with ever iteration, and it's down most weekends for upgrades.

    The VA benefits website is one of the more highly traffic'd government sites.  There is a tremendous amount of data being accessed on a daily basis.  It took several years and millions of man-hours of work to make the site what it is today - the preeminent benefits and healthcare website operated by the U.S. government at any level.

    Anyone with a web design or IT background could have told you that an ambitious undertaking such as was going to require an all hands on deck effort to make it seamless during the initial rollout.  And that it would probably be a trainwreck in the opening days.

    In my 30+ years in IT (including 3 SAP implementations), I have never - ever - experienced a seamless rollout.  You can load test the hell out of a system, spec it 9 ways from Sunday during the design phase, and it will still break on the first day.

    The concept of ACA is most assuredly not a failure because there are geek issues, and the geek issues will not cause a failure of the ACA (except in the fevered minds of Roger Ailes' crew).  Give it a few weeks.  The bugs will be worked out.

    UID: 14791 Join Date: 7/7/2004 Status: Lifetime member Mojo: nearly infinite Any questions?

    by Richard Cranium on Mon Oct 14, 2013 at 03:00:55 PM PDT

  •  Indoemative diary, thanks, (1+ / 0-)
    Recommended by:
    Boston to Salem

    Shut down due to Republican intransigence.

    by blueoregon on Mon Oct 14, 2013 at 03:09:22 PM PDT

  •  Many apps are websites (8+ / 0-)

    The dirty little secret about many iOS and Android "apps" is that they're an HTML5 code script inside a nice little Objective C or Java wrapper.

    That's because HTML is a lot easier for a non-programmer to update, so they can have a lower level employee update the content without having to recompile the code every time and push an annoying update onto the user.

    A modern SaaS website is a very complex interplay between a database and controllers that handle how the data is displayed and what functions the user can perform. We've come a long way from the days of a website being forms and fields and a background image.

    I agree with the diarist. As I said over in my own diary on this subject, all web services have crappy roll outs.    That's just the nature of web integrated software.

    The Cake is a lie. In Pie there is Truth. ~ Fordmandalay

    by catwho on Mon Oct 14, 2013 at 03:24:32 PM PDT

    •  It's not your great grandfather's web. :-) nt (1+ / 0-)
      Recommended by:

      Just another underemployed IT professional computer geek.

      by RhodeIslandAspie on Mon Oct 14, 2013 at 03:37:16 PM PDT

      [ Parent ]

    •  still doesn't (1+ / 0-)
      Recommended by:

      answer to why there wasn't a backup to the glitchy website?  There is no person to person way of getting me into system.  Why..would be my question. Why leave this important legislation and the importance for millions of Americans on a website?  Thats whats baffling.

    •  not all web-services have crappy rollouts (0+ / 0-)

      many do, yes, but I have personally delivered many, and they gleam from day one. Not 100% bug-free (nothing is bug free).

      ...agree with the diarist. .... all web services have crappy roll outs.    That's just the nature of web integrated software.
      It's not the nature of web-integrated software to be crappy, and I am talking about major stuff too, not the "my kitty photos" websites.

      It all depends on the team (the whole team, not just a few people). is not SaaS.

  •  I'm involved in infinitely simpler stuff than (10+ / 0-)

    this, and I've seen incredible screw ups, and I'm talking about not government, but private sector IT. I was involved in a brokerage where someone forgot to call the moving company that was to transport the equipment. I was involved in a retail store setup where everything we needed was sent - cash registers, servers, peripherals - except for a monitor to connect to the server. And without a monitor, I could not log in to the server to run the configuration scripts. And this working for a fortune 500 corporation. These were not brand new projects but fairly rote tasks that were pretty routine on both of these accounts. So let's not use this as an excuse to bash government, especially since this is government moving into some very new turf. Shit happens. Let's judge on what is learned from this, how quickly these issues are resolved.

    Just another underemployed IT professional computer geek.

    by RhodeIslandAspie on Mon Oct 14, 2013 at 03:32:57 PM PDT

  •  From what you say (5+ / 0-)

    The idea was to have a service application architecture backend that could respond either to state exchanges or to individuals logging in through the front-end of the federal exchange.

    In 2010, the idea was that there would likely be 50 state exchanges.  After the 2010 election as GOP governors started playing political games, more states moved to the federal exchange.  The last one decided was North Carolina, which notified the Feds in July 2013.

    That is a huge moving target of unexpected requirements even without the normal nonsense involved in federal IT projects.

    So there was sabotage from the GOP governors.  Given the sloppiness of some of the code, one should not discount political sabotage in the coding or in the hiring of coding staff.  There should be some investigation of this angle in order to discount it at any rate.

    The failures the first week are to be expected because there really are not good ways to model  large loads or to predict how large the load will be for this kind of application on startup.  And no doubt the fact of the governement shutdown increased interest in Obamacare and the size of the load.

    But the important point was the complexity of having a 50-state system that over a period of a couple of years shifted more states directly onto the federal front-end system and did not have closed requirements until some time after July 2013 when all of North Carolina's insurance plans and rules were available.

    50 states, 210 media market, 435 Congressional Districts, 3080 counties, 192,480 precincts

    by TarheelDem on Mon Oct 14, 2013 at 03:39:49 PM PDT

    •  Exactly! So many states punted to the feds and (1+ / 0-)
      Recommended by:

      took their sweet, sweet time doing it.

      if a habitat is flooded, the improvement for target fishes increases by an infinite percentage...because a habitat suitability index that is even a tiny fraction of 1 is still infinitely higher than zero, which is the suitability of dry land to fishes.

      by mrsgoo on Mon Oct 14, 2013 at 07:33:27 PM PDT

      [ Parent ]

  •  I worked with IT management in major (6+ / 0-)

    corporations as a Recruiter.

    I can tell you without fail that there has never been a bug free, error free roll out of any major program.  Ever.

    It doesn't happen.  There are always bugs to be worked out in any major program.  Also, don't forget the users/customers.  Half the time they just don't know how to navigate the site.

    It needs to be fixed, but I don't think we Democrats should start giving credence to every complaint coming from Republicans who are looking for any excuse to diss ACA.

    In the time it took Adam Lanza to reload, eleven children escaped. What if...

    by Sixty Something on Mon Oct 14, 2013 at 03:41:46 PM PDT

  •  Another commercial example, again using airlines: (6+ / 0-)

    When United merged with Continental (and the President of Continental became the President of the merged United Airlines), it was decided to use Continental's reservations and frequent flyer system for the merged airline instead of United's system (although they kept the name used by United for it). There were numerous problems during the first couple of weeks of this transition, especially for United agents not familiar with Continental's system. The problems were sorted out in a reasonable amount of time given the size of the merger, but for a short period of time there it was a mess. And now of course the system runs pretty well after the bugs and glitches have been identified and corrected and employees have received the proper training.

    Murphy's Law is just a fact of life, both big and small--all we can do is work to mitigate against it as much as we can. We can never eliminate Murphy's Law completely, but we can try to reduce Murphy's impact and try to make its inevitable effects a minor inconvenience instead of a major disaster. Nobody can write perfect code, especially when it starts to become quite complex--but pretty damn good code is a not unreasonable goal and expectation.

    But if the cause be not good, the king himself hath a heavy reckoning to make, ... there are few die well that die in a battle; ... Now, if these men do not die well, it will be a black matter for the king that led them to it; — Shakespeare, ‘Henry V’

    by dewtx on Mon Oct 14, 2013 at 03:43:00 PM PDT

  •  sabotage? (5+ / 0-)

    I would't be surprised if is being deliberately sabotaged— by a brute force denial of services attack, by hackers, or even by moles inside the project team.

    Obama's enemies are prepared to crater the global financial system to stop ObamaCare.  They have shut down the federal government.  They have totally crippled the U.S. House of Representatives and also undermined the integrity of the Senate.  Crashing a web site seems like child's play in comparison.

    The only thing which makes me doubt this theory is the fact Obama's enemies are stupid.  Crashing a web site may be child's play, but it is the play of bright children.

    •  Same thought here (0+ / 0-)

      There are plenty of right wing h4XX0r5 with sufficient ability to impede the website launch.  

      More probable IMO would be a deliberate sabotage in the development phase.  Several developers I've spoken with have all noted that some of the poor coding choices were inexplicable.  What do we know of the private contractor which developed HealthCare.Gov?  Are they owned or run by wingnuts?  Is the hiring process rigorous enough to filter out any wingnut moles?  Are there enough checks and balances in the company organization to negate the effects of saboteurs?  

      Certainly much of the blame lies with the government's oversight of the development process, and with the unexpected demand.  But the radical right is fully capable of sabotage - in fact it's their line of work when it comes to government.  It would be wise for the Obama admin to assign an IG to investigate the development process and determin if there was any foul play.

      "When I was an alien, cultures weren't opinions" ~ Kurt Cobain, Territorial Pissings

      by Subterranean on Mon Oct 14, 2013 at 10:10:13 PM PDT

      [ Parent ]

  •  The problem seems to be affecting the states that (4+ / 0-)
    Recommended by:
    Sherri in TX, ybruti, mrsgoo, amsterdam

    chose not to set up their own exchanges, since they are the ones where people have to go directly though the federal system. This is absolutely delicious. The red controlled states whose leaders are always telling us how wonderful it is that the states can be 50 laboratories to test new ideas, are suffering, because they declined to let their states be such laboratories. And that states that eagerly set up their own are doing quite well, thank you. Rhode Island's site is working quite well.

    Just another underemployed IT professional computer geek.

    by RhodeIslandAspie on Mon Oct 14, 2013 at 03:49:39 PM PDT

    •  think it is (0+ / 0-)

      effecting all the exchanges as the IRS, HHS, Homeland Security all have to verify your info....state exchanges I believe are also getting hung up in this part of the process...the backend architecture is screwed up...probably a lot to do with legacy systems in place...this is going to take awhile to fix....

    •  I signed up in Ohio (2+ / 0-)
      Recommended by:
      Jakeston, amsterdam

      Which uses the national exchange. I selected a plan and thus completed enrollment.

      A few days ago I got an information packet from the selected insurance company (CareSource). It simply said that coverage would start Jan. 1, 2014 if I send in the first payment by Oct. 31st. There were several pages outlining the plan benefits and conditions.

      Reports of the non-functionality of the web site are greatly exaggerated.

  •  Tipped and Rec'd. Great discussion here. (5+ / 0-)

    One suggestion: I read diary as an ex-fed so understood all, but some may not know RFP = request for proposal, for an example of acronym needing spell-out. The gov't publishes a good-sized book on its acronyms because use is so extensive that is can bewilder even pros. Thanks to many IT professionals commenting, but could I use more spell-outs there, too. Thanks all for best diary and discussion I read today.  

    The trouble ain't that there is too many fools, but that the lightning ain't distributed right. Mark Twain

    by BlueMississippi on Mon Oct 14, 2013 at 05:58:32 PM PDT

  •  Very nice diary (3+ / 0-)
    Recommended by:
    BlueMississippi, mrsgoo, amsterdam

    My husband has worked in IT for years at various jobs. For more than ten years, I've listened to him complain about the difficulties of putting together a project because the different departments don't know how to communicate, and I've also listened to the troubleshooting he's been involved in after the rollout of every single project he's ever done. He's never, ever had a smooth rollout. In fact, the company he works at now plans for that when they are determining man hours on the project.

    On top of that, I'm a gamer and have definitely had my share of experiences with launch failures. It just comes with the territory.

    Now if is still performing assbackwards in a month, that's a different story. Until then, meh.

    Time is of no account with great thoughts, which are as fresh to-day as when they first passed through their authors' minds ages ago. - Samuel Smiles

    by moviemeister76 on Mon Oct 14, 2013 at 06:53:59 PM PDT

  •  Another reality-based diary on the ACA. (4+ / 0-)

    Getting a little sick of the hysterical "worst-case scenarios."

    Tipped and recommended wholeheartedly.

    It's here they got the range/ and the machinery for change/ and it's here they got the spiritual thirst. --Leonard Cohen

    by karmsy on Mon Oct 14, 2013 at 07:09:49 PM PDT

  •  the reason some (me included) say "Google..etc" (0+ / 0-)

    is that Google and Amazon have people that know how to design and build large-scale high-capacity and so-called "scalable' systems.

    The approach wherein significant input data is 'lost' (meaning my application with a whole lot of input-data is unavailable to me) is simply unforgivable of any company who is paid to build a system whose purpose is predicated on gathering of data.

    I tried to create an account, and after a looooong hanging no-response, it simply failed and said "system unavailable at this time" or similar. Step 1 in creating an account should be to email a user-ID; on a system of this scale, the user should be informed "you have X hours to validate this userID and complete input before the data is purged" (x should be chosen to balance user-frustration against pranksters and ill-wishers who want to overwhelm with junk data. Also absent in any of my input-forms was something Captcha-like to ensure that Freedumbworks or some other clown-show is not trying DoS attacks with scripted junk-registrations.

    No, folks - this is implementation-team incompetence plain and simple. Of course Fox and friends will spin this as government malfunction, but anyone from the software line of business can tell many stories of total failure when government was nowhere near the software project.

    I design large-scale systems for very high transaction-rates (2000 TPS) - multi-site distributed clusters, high-availability systems with load-balancing etc. etc., and not losing data (particularly if it is user-input, form-submitted is not hard if you have a clue what you're doing. Most have no clue, sadly.

    Fact: large amounts of IT effort in the USA is outsourced/offshored, regardless of the rules of law - the rules are simply flouted by setting up shell companies, on-shoring essentially off-shore resources and putting them up in ultra-cheap accomodation with very low per-diem allowances, then throwing them back where they came from before temp work-visas expire

    The failure is simply a result of

    • people who don't know how to do their freaking jobs.  along list, starting with business-analyst/requirements-gatherers, specification-writers, so-called architects who obviously fell down on the job, technical team-leads who don't know how to lead teams, review code, review test-results, and crappy developers who in many cases have no interests beyond getting to the end of the day
    • other people who know very well how to send invoiced and take the payments without regard for quality or reliability - i.e. the profiteers and their middle-management enablers. These are often people who have no clue whatsoever about the process of designing and delivering software

    This lousy performance has nothing to do with government (as you and many already correctly point out ); incompetence knows no boundaries. Anyone who says government is to blame for this failure knows nothing about software - total, abject failure and massive cost-overrun are common in this business - government-related or not

    •  From a purely technical perspective, I agree (0+ / 0-)

      Although again, in terms of the systems to which you apparently need to integrate in this case, they could be just as much of the problem as the Healthcare.Gov application itself.

      But from a procurement perspective, unfortunately the way the government works, and the way those companies work, it would have been extremely difficult and expensive to get the kind of talent you are talking about in place to do this work.  Unless they had donated the man hours for free, which would raise its own complications.

      -9.00, -5.85
      Quite an experience to live in fear, isn't it? That's what it is to be a slave.

      by Wintermute on Tue Oct 15, 2013 at 02:38:52 AM PDT

      [ Parent ]

  •  It's going better than the Sim City rollout (0+ / 0-)

    You've got to give them that much.

  •  sabatage (0+ / 0-)

    Has anyone done any Tea Party background checks to find out who is working for the Affordable Health Care.  Seems more than a little strange that so many people are finding problems getting on the sight.  Way too strange.

Sylv, TXdem, askew, Trendar, NYmom, madmsf, TX Unmuzzled, jancw, Wee Mama, amsterdam, whenwego, boadicea, Aquarius40, roses, pollbuster, denig, Boston to Salem, splashy, Alna Dem, psnyder, emmasnacker, crankypatriot, laughingriver, Catte Nappe, riverlover, Sophie Amrain, monkee, ybruti, WisVoter, tomjones, Chinton, chimene, Erik the Red, marathon, Blu Gal in DE, turdraker, kkjohnson, Alan Arizona, kathny, lgmcp, elliott, BachFan, edwardssl, emeraldmaiden, raptavio, Dvalkure, Shakludanto, DarkestHour, Spock36, SingerInTheChoir, BlueMississippi, doingbusinessas, Clive all hat no horse Rodeo, blueoregon, kurt, The Knute, FlamingoGrrl, Habitat Vic, karmsy, deepeco, Librarianmom, dizzydean, nuketeacher, mathGuyNTulsa, roycej, mconvente, VelvetElvis, GAS, Sixty Something, Involuntary Exile, salamanderanagram, Lujane, tofumagoo, pamelabrown, smartdemmg, bluesheep, jorogo, mattc129, statsone, Methinks They Lie, mkor7, virginwoolf, Keith Pickering, moviemeister76, astral66, Larsstephens, BlueOak, Railfan, parse this, serendipityisabitch, catwho, nickrud, Betty Pinson, MsGrin, petesmom, Quantumlogic, renbear, misshelly, MidwestTreeHugger, ban nock, La Gitane, UtahLibrul, asterkitty, Araguato, OhioNatureMom, CoExistNow, sethtriggs, LSmith, worldlotus, zapus, VTCC73, doct, freerad, ratcityreprobate, SoCalSal, CoyoteMarti, bassinduo, ahumbleopinion, DrLori, FloridaSNMOM, Mr Robert, peachcreek, Arahahex, lunachickie, james321, jusjtim35, newinfluence, smokey545, Witgren, SheilaKinBrooklyn, MemaW, howabout, RhodeIslandAspie

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site