Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More On September 5th, 2024, Matt Shumer, co-founder and CEO of the startup Hyperwrite AI (also known as OthersideAI) took to the social network X to post the bombshell news that he had fine-tuned a version of Meta’s open source Llama 3.1-70B into an even more performant large language model (LLM) known as Reflection 70B — so performant, in fact, based on alleged third-party benchmarking test results he published, that it was “the world’s top open-source model,” according to his post. I’m excited to announce Reflection 70B, the world’s top open-source model. Trained using Reflection-Tuning, a technique developed to enable LLMs to fix their own mistakes. 405B coming next week – we expect it to be the best model in the world. Built w/ @GlaiveAI. Read on ⬇️: pic.twitter.com/kZPW1plJuo — Matt Shumer (@mattshumer_) September 5, 2024 However, shortly after its release, third-party evaluators in the AI research and hosting community struggled to reproduce the claimed results, leading to accusations of fraud. Researchers cited discrepancies between the announced benchmark results and their independent tests, sparking a wave of criticism on social platforms such as Reddit and X. In response to these concerns, Shumer pledged he would conduct a review of the issues alongside Sahil Chaudhary, founder of Glaive, the AI startup whose synthetic data Shumer claimed he had trained Reflection 70B on — and which he later revealed to have invested what he called a small amount into. Now, nearly a month later, Chaudhary last night released a post-mortem report on his Glaive AI blog about the Reflection 70B model and published resources for the open-source AI community to test the model and his training process on their own. He says while he was unable to reproduce all of the same benchmarks, he “found a bug in the initial code,” resulting in several results appearing higher than what he has found on recent tests of Reflection 70B. However, other benchmark results appear higher than before — adding to the mystery. On September 5th, @mattshumer_ announced Reflection 70B, a model fine-tuned on top of Llama 3.1 70B, showing SoTA benchmark numbers, which was trained by me on Glaive generated data. Today, I’m sharing model artifacts to reproduce the initial claims and a post-mortem to address… — Sahil Chaudhary (@csahil28) October 2, 2024 As Chaudhary wrote in the post: “There were a lot of mistakes made by us in the way we launched the model, and handled the problems reported by the community. I understand that things like these have a significant negative effect on the open source ecosystem, and I’d like to apologize for that. I hope that this adds some clarity to what happened, and is a step in the direction of regaining the lost trust. I have released all of the assets required to independently verify the benchmarks and use this model.“ Sharing model artifacts To restore transparency and rebuild trust, Chaudhary shared several resources to help the community replicate the Reflection 70B benchmarks. These include: Model weights: Available on Hugging Face, providing the pre-trained version of Reflection 70B. Training data: Released for public access, enabling independent tests on the dataset used to fine-tune the model. Training scripts and evaluation code: Available on GitHub, these scripts allow for reproduction of the model’s training and evaluation process. These resources aim to clarify how the model was developed and offer a path for the community to validate the original performance claims. Reproducing the benchmarks In his post-mortem, Chaudhary explained that a major issue with reproducing the initial benchmark results stemmed from a bug in the evaluation code. This bug caused inflated scores in certain tasks, such as MATH and GSM8K, due to an error in how the system handled responses from an external API. The corrected benchmarks show slightly lower, but still strong, performance relative to the initial report. The updated benchmark results for Reflection 70B are as follows: MMLU: 90.94% GPQA: 55.6% HumanEval: 89.02% MATH: 70.8% GSM8K: 95.22% IFEVAL: 87.63% Compare that to the originally stated performance of: MMLU: 89.9% GPQA: 55.3% HumanEval: 91% MATH: 79.7% GSM8K: 99.2% IFEVAL: 90.13% Although the revised scores are not as high as those initially reported, Chaudhary asserts that they are more accurate reflections of the model’s capabilities. He also addressed concerns about dataset contamination, confirming that tests showed no significant overlap between the training data and benchmark sets. Reflecting on a hasty release Chaudhary admitted that the decision to release Reflection 70B was made hastily, driven by enthusiasm for the model’s performance on reasoning-based tasks. He noted that the launch lacked sufficient testing, particularly regarding the compatibility of the model files, and that he and Shumer had not verified whether the model could be easily downloaded and run by the community. “We shouldn’t have launched without testing, and with the tall claims of having the best open-source model,” Chaudhary wrote. He also acknowledged that more transparency was needed, especially regarding the model’s strengths and weaknesses. While Reflection 70B excels at reasoning tasks, it struggles in areas like creativity and general user interaction, a fact that was not communicated at launch. Clarifying API confusion One of the more serious accusations involved the suspicion that the Reflection 70B API was simply relaying outputs from Anthropic’s Claude model. Users reported strange behavior in the model’s outputs, including responses that seemed to reference Claude directly. Chaudhary addressed these concerns, explaining that although some of these behaviors were reproducible, he asserts there was no use of Claude APIs or any form of word filtering in the Reflection 70B model. He reiterated that the API was run on Glaive AI’s compute infrastructure, and Matt Shumer had no access to the code or servers used during this period. Looking ahead In closing, Chaudhary emphasized his commitment to transparency and expressed his hope that this post-mortem and the release of model artifacts will help restore trust in the project. He also confirmed that

On June 10, 2024, Apple entered a new era of intelligence. At their annual Worldwide Developers Conference (#WWDC24), Apple unveiled how artificial intelligence will be embedded into their ecosystem and announced several enhanced features for iOS, iPadOS, and MacOS. These new smarter features will empower users through the use of AI. This year’s WWDC marks a pivotal moment for Apple, being one of the most significant events in recent years. Since the introduction of the iPhone, the iPad, and the Apple watch, Apple has been the undisputable leader in terms of the user experience, and dominated sales of smartphones, tablets and smartwatches in value terms. Over the last 17 years, Apple has not faced any other disruptive technology as potentially detrimental for its business and its future as AI, if not tackled in the right way. WWDC 2024 offered Apple and its CEO, Tim Cook, an opportunity to demonstrate how the company will lead in making AI a transformative, advanced, and intelligent experience for their users. With smartphone, tablet and PC sales slowing down due to high adoption rates and offering just incremental improvements from the previous version of devices, Apple needs to reignite consumers excitement to encourage more frequent upgrades. We all remember the long queues outside Apple stores when a new iPhone was launched! The excitement generated by past product launches is critical to Apple’s business and brand. AI Will Enable Apple To Offer Unique and Intelligent Features, Experiences and Services to Their Customers. However, the key question remains: will these announcements be enough to secure Apple’s leading position? Many competitors (from phones to PCs vendors) have already revealed their AI strategies and devices. How can Apple stand out and how can a partnership with OpenAI help? AI-Enabled Devices Will Be the Fastest Growing Segment in for Smartphones and PCs. IDC forecasts that AI Smartphones to reach 170 million units in 2024, and AI PCs to account for nearly 60% of all PC shipments by 2027. Historically, the App Store has been at the heart of the value proposition of Apple’s iPhone and the iPad. The slogan, “there’s an App for that” became iconic. However, AI is set to “kill” the apps and a new era is about to begin for Apple and the industry. Previously, the more apps the smartphone (and the iPad) was able to run, and the more powerful the apps, the better the smartphone (or iPad). Today, the smartphone is not ‘smart’ anymore. With AI, the fewer apps needed and the more a phone can use data contextually to assist the user, the better the phone will be. This will revolutionize the user experience, requiring an operating system able to learn from all devices and services in Apple’s ecosystem. Apple’s full control of its ecosystem, hardware and software, and seamless integration gives it a significant advantage over competitors. This is particularly important when the experience will be defined by how well the phone “knows” the user, which can only be achieved by this seamless integration, while making sure the personal information and data remains private. Although Apple is not the first to offer AI-enabled features, it is in its DNA to offer the perfect experience. AI will not just offer the best experience, but the most powerful, disruptive, personalized and private, and that’s exactly what Apple showed today with Apple Intelligence. While AI-generated emojis will grab the headlines, Apple showcased how AI will empower users in several ways, particularly around productivity, education and entertainment. These are the areas where Apple already excels with its range of services and applications. As User Interaction Shifts From Opening an App To Asking the Phone, Siri Will Be the Glue. This killer feature will attract consumers to AI enabled devices. Siri will offer a unique way to interact with all Apple devices using our voice in a more personalized and conversational manner. It will become a true digital assistant, contextually understanding various aspects of users’ lives and providing human-like responses by accessing data across all devices to provide the right response to the users’ needs. This conversational digital assistant fully integrated with all devices will be a game changer, and I believe it will offer a compelling reason for many to upgrade their iPhones in the years to come. Apple’s strength lies in the breath of its portfolio and its ecosystem. For consumers, it does not matter what an AI-enabled iPhone is, or how many TOPs it can run. What matters is how well a phone can assist users in managing their lives. By reducing the need to navigate multiple apps, Apple’s AI-enabled devices will offer a smoother, more intuitive and smarter experience. This marks the beginning of a new era for Apple and for their users. For more insights and views, watch this conversation with Tom Mainelli at the Apple Park. Learn what matters most to your customers with IDC’s AI Use Case Discovery Tool—find out more. source

Digital business is just standard business in 2024. Companies striving to participate in the digital economy are looking to invest in technology that fits the needs of their company size, employee personas and functions, industry verticals, and current level of technology maturity. Vendors hoping to sell and implement digital technologies need to consider all these factors when meeting with a potential customer. Segmenting and grouping customers by all these factors can help technology vendors provide the best messaging and service. At IDC, we do research that considers the impacts of all these factors, and that gives us insight behind the tech buying curtain into the needs and wants of buyers at all stages of their digital journeys. The Small and Medium Business Market In the recent IDC Webcast, “Behind the Tech Buying Curtain: What Vendors Need to Know,” Katie Evans, IDC Senior Research Director for Worldwide Small and Medium Business Markets, said there are three key topics keeping SMB tech buyers up at night: AI/automation, macroeconomic woes, and heightened security concerns largely fueled by AI, remote work, and the moving to the Cloud. The forward-looking investment priorities for SMBs are all automation and AI related—process automation, connectivity automation, AI (non-GenAI), and GenAI, according to IDC’s Worldwide Small and Medium Business Survey. Rising energy prices highlight SMB’s top macroeconomic woes in that survey, while implementing technology securely is the largest technology challenge. Macro Factors Impacting Investment IDC’s Digital Economy Strategies research, led by Research Analyst Elisabeth Clemmons, focuses in part on the macro landscape impacting businesses, even those focused in specific countries or localities. Impactful events and trends include skills shortages, inflation, supply-chain constraints, energy crises, tensions between countries and war, and elections around the world. These events impact technology buying patterns, causing companies to be more cautious with their investments. AI is the technology most likely to persevere and be prioritized in the face of these macro impacts, with AI spending, the AI provider supply chain, and the economic stimulus among AI adopters is projected by IDC’s Macroeconomic Center of Excellence to be 3.5% of global GDP by 2030. The heightened impact of and investment in AI is contributing to two more macro trends impacting businesses: digital regulation and potential raw materials shortages. Data and AI is a top regulatory target with governments taking diverse approaches, while the sheer amount of data in the world is expected to more than triple by 2028—necessitating over 50 times the current annual production levels of neodymium and other critical raw materials. C-Suite Leaders Increase Focus on AI Despite any concerns and challenges that AI my pose, CEOs are scaling AI initiatives, dedicating budgets to AI, and ensuring AI projects receive greater visibility, according to Nupur Singh-Adley, Research Manager leading IDC’s C-Suite Tech Agenda research. As C-suite leaders focus on building digital businesses buoyed by AI technology, they are also mindful of managing risk, including heightened cybersecurity threats and regulations. This is leading to prioritized spending on security, risk, and compliance technologies and greater scrutiny of cybersecurity and risk management at the board level. Along with considering risk, CEOs are also prioritizing responsible AI while building digital businesses that are focused on trust, critical evaluation of tech vendors, and sustainability. Digital Capability Perception vs. Reality While companies of all sizes are facing challenges both internally and externally and they digitize, it’s important for technology vendors to consider that many are also overestimating their digital capabilities. In IDC’s Digital Business and AI Transformation Strategies research, we conduct an annual Digital Business Scorecard that measures the digital capabilities of businesses in four areas: Digital Business Models, Data, Operational Processes, and Organization. Overall, when asked to assess themselves, 41% of those surveyed in IDC’s Digital Executive Sentiment Survey believed they were a “mostly digital business.” However, when applying our Digital Business Scorecard methodology, which correlates investments and strategies to business outcomes, only 11% of that same survey sample were categorized as “Digital Business Leaders.” Digital Business Leaders are taking holistic digital strategies and have implemented world-class technology. They view data as a top AI priority while also expanding its use in core operational processes. They have a digital technology architecture that is in lockstep with IT strategy and are focused on improving recruiting and employee engagement. These are the standards that many of those companies that think they are digital aren’t quite achieving—often finding themselves bogged down by inferior technology, not focusing on capitalizing on the value of their data, not looking to automate and standardize processes, and not taking the view of technology as a competitive advantage. The Important Role of Technology Vendors in the Digital Economy Technology vendors are key to digital business and AI transformation strategies—and many companies need vendor expertise and guidance to help them become digital business leaders. Vendor messaging to these companies can be challenging—but it helps to know how company size, macroeconomic impacts, C-suite leadership, and digital capability and perception impacts buying conversations. It’s helpful for vendors to segment their customers, and specialize their strategy and messaging to those individual segments. That’s beneficial for all parties and will help companies succeed in the digital economy. Learn what matters most to your customers with IDC’s AI Use Case Discovery Tool—find out more. source

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Is the future of written media — and potentially imagery and videos, too — going to be primarily surfaced to us through ChatGPT? It’s not out of the question at the rate OpenAI is going. At the very least, the $157-billion dollar valued AI unicorn — fresh off the launch of its new Canvas feature for ChatGPT and a record-setting $6.6 billion fundraising round — is making damn well sure it has most of the leading U.S. magazine and text-based news publishers entered into content licensing agreements with it. These enable OpenAI to train on, or at least serve up, vast archives of prior written articles, photos, videos and other journalistic/editorial materials, through ChatGPT, SearchGPT and other AI products, potentially as truncated summaries. The latest major American media firm to join with OpenAI is Hearst, the eponymous media company famed for its “yellow journalism” founder William Randolph Hearst (who helped beat the drum for the U.S. to enter the Spanish-American War as well as demonized marijuana, and was memorably fictionalized by Citizen Kane‘s Charles Foster Kane) which is now perhaps best known as the publisher of Cosmopolitan, the sex and lifestyle magazine aimed at young women, as well as Esquire, Elle, Car & Driver, Country Living, Good Housekeeping, Popular Mechanics and many more. In total, Hearst operates 25 brands in the U.S., 175 websites and more than 200 magazine editions worldwide, according to its media page. However, OpenAI will be specifically surfacing “curated content” from more than 20 magazine brands and over 40 newspapers, including well-known titles such as Cosmopolitan, Esquire, Houston Chronicle, San Francisco Chronicle, ELLE, and Women’s Health. The content will be clearly attributed, with appropriate citations and direct links to Hearst’s original sources, ensuring transparency, according to the brands. “Hearst’s other businesses outside of magazines and newspapers are not included in this partnership,” reads a release jointly published on Hearst’s and OpenAI’s websites. It’s unclear whether or not the company will be training its models specifically on Hearst content — or merely piping said content through to end users of ChatGPT and other products. I’ve reached out to an OpenAI spokesperson for clarity and will update when I hear back. Hearst now joins the long and growing list of media publishers that have struck content licensing deals with OpenAI. Among the many that have forged deals with OpenAI include: These partnerships represent OpenAI’s broader ambition to collaborate with established media brands and elevate the quality of content provided through its AI systems. With Hearst’s integration, OpenAI continues to expand its network of trusted content providers, ensuring users of its AI products, like ChatGPT, have access to reliable information across a wide range of topics. What the executives are saying it means Jeff Johnson, President of Hearst Newspapers, emphasized the critical role that professional journalism plays in the evolution of AI. “As generative AI matures, it’s critical that journalism created by professional journalists be at the heart of all AI products,” he said, underscoring the importance of integrating trustworthy, curated content into these platforms. Debi Chirichella, President of Hearst Magazines, echoed this sentiment, noting that the partnership allows Hearst to help shape the future of magazine content while preserving the credibility and high standards of the company’s journalism. These deals signal a growing trend of cooperation between tech companies and traditional publishers as both industries adapt to the changes brought about by advances in AI. While OpenAI’s partnerships offer media companies access to cutting-edge technology and the opportunity to reach larger audiences, they also raise questions about the long-term impact on the future of publishing. Some critics argue that licensing content to AI platforms could potentially lead to competition, as AI systems improve and become more capable of generating content that rivals traditional journalism. I myself, as a journalist whose work was undoubtedly scraped and trained by many AI models (and used for lots of other things of which I had no control over or say in), voiced my own hesitation about media publishers moving so quickly to ink deals with OpenAI. These concerns were amplified in recent legal actions, such as the lawsuit filed by The New York Times against OpenAI and Microsoft, alleging copyright infringement in the development of AI models. The case remains in court for now, and NYT remains one of an increasingly few holdouts who have yet to settle with or strike a deal with OpenAI to license their content. Despite these concerns, publishers like Hearst, Condé Nast, and Vox Media are actively embracing AI as a means of staying competitive in an increasingly digital landscape. As Chirichella pointed out, Hearst’s partnership with OpenAI is not only about delivering their high-quality content to a new audience but also about preserving the cultural and historical context that defines their publications. This collaboration, she said, “ensures that our high-quality writing and expertise, cultural and historical context and attribution and credibility are promoted as OpenAI’s products evolve.” For OpenAI, these partnerships with major media brands enhance its ability to deliver reliable, engaging content to its users, aligning with the company’s stated goal of building AI products that provide trustworthy and relevant information. As Brad Lightcap, COO of OpenAI, explained, bringing Hearst’s content into ChatGPT elevates the platform’s value to users, particularly as AI becomes an increasingly common tool for consuming and interacting with news and information. source