Here are the bullets and worse of all it’s a software product that brought the system to its knees. Did they not test before hand
- Washington state regulators said the system was back online at 5:50 a.m. PT and notified cannabis business licensees shortly thereafter.
- That’s after the Washington State Liquor and Cannabis Board sent a release Tuesday afternoon that an error in the most recent Leaf Data Systems software release, provided by Denver-based firm MJ Freeway, was “causing a breakdown” and preventing wholesale firms from conducting business transactions within the system.
- Testing lab results were not “associating” with product information, and therefore, producers and processors were unable to create manifests to buy and sell products. This caused a disruption in the cannabis supply chain, with growers unable to get their product to processors and producers unable to supply retailers.
Also we’ve been alerted to the following WSLCB – Board Caucus (July 16, 2019) minutes which make interesting reading
We preface the minutes with this highlighted section
Here are some observations from the Tuesday July 16th Washington State Liquor and Cannabis Board (WSLCB) Board Caucus.
My top 3 takeaways:
- Board Chair Jane Rushford briefly noted “there were some considerations” which were keeping Deputy Director Megan Duffy from providing an update on the traceability system re-launch; the agency and its vendor MJ Freeway took Leaf Data Systems offline later that afternoon and worked through the night to patch the system in uncertain and unsettling circumstances.
- Following a planned two-day outage spanning Sunday and Monday, the agency announced the launch of release 1.37.5 of Washington State’s version of MJ Freeway’s Leaf Data Systems on Tuesday morning at 7am. Reports of unanticipated problems began surfacing shortly thereafter.
- After the 10am PT Board Caucus, the agency distributed an announcement at 2pm PTconfirming “an error in Leaf Data Systems software release 1.37.5 that is causing a breakdown in the ability for licensees to do wholesale business transactions.” The system was taken offline as “MJF is currently isolating the code error that is causing the disassociation and preparing a fix.” To facilitate business-to-business (B2B) transactions, licensees were encouraged to use an online manifest form – the same form prepared for licensee usage during the contingency reporting period necessitated by MJ Freeway’s failure to hit their first deadline on November 1, 2017.
- A planned 3:30pm PT “Post Go-Live Check-In Call” with the software integrators and interested parties was repurposed into a forum for deliberation about six primary issues encounteredwhose causes were not yet identified. The most problematic issue was a mysterious recurring disassociation of lab testing results from licensee inventory items. WSLCB Chief Information Officer Mary Mueller said roughly 4 million lab results were orphaned, although the underlying data was not lost. The agency was preparing for two options: successful identification and repair of the root cause of the issues, or rollback to the previous software release including erasure of traceability data collected by the state on Monday.
- At 4:15pm PT, the agency convened with their vendor MJ Freeway to learn about progress towards identification of root causes. A hastily arranged call with the software integrators at 8pm PT revealed the agency was still at sea and the vendor still had no solutions to offer.
- On Wednesday morning at 12am PT, the agency released a carefully worded description of two scenarios for bringing the traceability system back online. 3am PT was declared as the decision point for the agency to choose a path forward (or backward).
- At 5:30am PT, the third-party software providers were informed that a mitigation had been achieved and tested to the agency’s satisfaction. At 6am PT, an announcement was broadcast by the WSLCB that the industry would be moving forward with a re-launch of a patched version of release 1.37.5 inclusive of Monday’s traceability data.
- Leaf Data Systems version 1.37.5 has been particularly challenging for the vendor and the state to achieve. During the course of this latest incident, Cannabis Observer has witnessed calls for a class-action lawsuit against the state; concerns about reversion to a spreadsheet-based contingency reporting paradigm; and indications that legislators and the Office of the Governor will be asked to force the WSLCB to cancel the contract with MJ Freeway. At a minimum, the WSLCB will be negotiating and collecting liquidated damages from MJ Freeway, a stipulation established by former Deputy Director Pat Kohler when the vendor first failed to deliver release 1.37.5 in November 2018. Unfortunately, it’s worth noting that while the agency’s decision to move forward was likely a better path than attempting to shift an entire industry backwards in time – it remains to be seen how well the patch was executed and what other unanticipated issues may lurk in MJ Freeway’s closed, proprietary source code.