As retailers grapple with challenges from complex supply chains to skyrocketing return costs, staying ahead requires bold innovation and smart partnerships. Our new report, The 2025 NRF Innovators: Retail-Focused Tech Companies To Watch, produced in partnership with the National Retail Federation (NRF) Innovation Advisory Committee, is an overview of 50 companies tackling those very challenges. These companies are exhibiting as part of the 2025 NRF Big Show Innovators Showcase this coming week. Who are the 2025 NRF Innovators, and how are they chosen? The NRF Innovation Advisory Committee (IAC), a group of over 20 industry leaders, venture capitalists, and technology accelerators (of which Forrester is a member), selected and invited the companies as participants of the 2025 NRF Innovators Showcase. The IAC evaluates companies based on four key criteria: Problem relevance. Does the technology solve a critical issue for retailers? Significance. How widespread and impactful is the challenge that is being addressed? Product-to-market fit. Has the solution proven its value through partnerships and pilots? Scalability. Is the company ready to deliver at scale across the industry? What are some of the retail challenges that these companies tackle? Retail contributes $5 trillion annually to the US economy and employs over 50 million people, yet the industry faces several hurdles: Complex supply chains. Rising costs and global trade shifts are disrupting the flow of goods, leading to a need for more agile and efficient solutions. In-store challenges. With over 70% of retail sales still happening in physical stores, retailers must balance operational efficiency with delivering engaging customer experiences. High customer expectations. Shoppers now expect seamless, personalized interactions across all channels and touchpoints. Costly reverse logistics. Managing returns, which cost retailers nearly $890 billion annually, remains a significant strain on profitability. This report showcases technologies addressing these challenges head on, offering a roadmap for resilient and profitable retail operations in four areas (see the full list of 2025 NRF Innovators in the table below): Revolutionizing supply chains. Streamlined supply chains are nonnegotiable for modern retailers. Companies in this group are transforming supplier collaboration, simplifying global sourcing, and accelerating supply chain communication with rapid electronic data interchange onboarding, among other areas. Transforming in-store operations. Physical stores remain the retail backbone, but they need smarter tools. Innovations such as smart carts and enhanced security systems enhance operational efficiency and customer engagement. These technologies bridge the gap between digital and physical retail spaces. Personalizing customer engagement. Personalization drives customer loyalty and sales. Tools like content creation and management solutions, shoppable video content, and digital display insights, among others, enable personalized experiences, both online and in the store. Streamlining payments and returns. Efficient payment and return processes are key to profitability. Think solutions such as optimizing reverse logistics, minimizing payment declines, integrating payment methods, and enhancing last-mile delivery. If you’re at the NRF Big Show, do stop by the 2025 NRF Innovators Showcase in the River Pavilion at the Javits Center from January 12 to 14 to meet these companies. NRF members and Forrester clients can download the report from their respective websites to see more detailed descriptions of each of these companies. The 2025 NRF Innovators source

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More The industry’s push into agentic AI continues, with Nvidia announcing several new services and models to facilitate the creation and deployment of AI agents.  Today, Nvidia launched Nemotron, a family of models based on Meta’s Llama and trained on the company’s techniques and datasets. The company also announced new AI orchestration blueprints to guide AI agents. These latest releases bring Nvidia, a company more known for the hardware that powers the generative AI revolution, to the forefront of agentic AI development. Nemotron comes in three sizes: Nano, Super and Ultra. It also comes in two flavors: the Llama Nemotron for language tasks and the Cosmos Nemotron vision model for physical AI projects. The Llama Nemotron Nano has 4B parameters, the Super 49B parameters and the Ultra 253B parameters.  All three work best for agentic tasks including “instruction following, chat, function calling, coding and math,” according to the company. Rev Lebaredian, VP of Omniverse and simulation technology at Nvidia, said in a briefing with reporters that the three sizes are optimized for different Nvidia computing resources. Nano is for cost-efficient low latency applications on PC and edge devices, Super is for high accuracy and throughput on a single GPU and Ultra is for highest accuracy at data center scale.  “AI agents are the digital workforce that will work for us and work with us, and so the Nemotron model family is for agentic AI,” said Lebaredian.  The Nemotron models are available as hosted APIs on Hugging Face and Nvidia’s website. Nvidia said enterprises can access the models through its AI Enterprise software platform.  Nvidia is no stranger to foundation models. Last year, it quietly released a version of Nemotron, Llama-3.1-Nemotron-70B-Instruct, that outperformed similar models from OpenAI and Anthropic. It also unveiled NVLM 1.0, a family of multimodal language models.  More support for agents AI agents became a big trend in 2024 as enterprises began exploring how to deploy agentic systems in their workflow. Many believe that momentum will continue this year.  Companies like Salesforce, ServiceNow, AWS and Microsoft have all called agents the next wave of gen AI in enterprises. AWS has added multi-agent orchestration to Bedrock, while Salesforce released its Agentforce 2.0, bringing more agents to its customers.  However, agentic workflows still need other infrastructure to work efficiently. One such infrastructure revolves around orchestration, or managing multiple agents crossing different systems.  Orchestration blueprints  Nvidia has also entered the emerging field of AI orchestration with its blueprints that guide agents through specific tasks.  The company has partnered with several orchestration companies, including LangChain, LlamaIndex, CrewAI, Daily and Weights and Biases, to build blueprints on Nvidia AI Enterprise. Each orchestration framework has developed its own blueprint with Nvidia. For example, CrewAI created a blueprint for code documentation to ensure code repositories are easy to navigate. LangChain added Nvidia NIM microservices to its structured report generation blueprint to help agents return internet searches in different formats.  “Making multiple agents work together smoothly or orchestration is key to deploying agentic AI,” said Lebaredian. “These leading AI orchestration companies are integrating every Nvidia agentic building block, NIM, Nemo and Blueprints with their open-source agentic orchestration platforms.” Nvidia’s new PDF-to-podcast blueprint aims to compete with Google’s NotebookLM by converting information from PDFs to audio. Another new blueprint will help build agents to search for and summarize videos.  Lebaredian said Blueprints aims to help developers quickly deploy AI agents. To that end, Nvidia unveiled Nvidia Launchables, a platform that lets developers test, prototype and run blueprints in one click.  Orchestration could be one of the bigger stories of 2025 as enterprises grapple with multi-agent production.  source

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Nvidia and its partners have launched agentic AI blueprints to automate work for enterprises. Developers can now build and deploy custom AI agents that can reason, plan and take action with Nvidia AI blueprints that include Nvidia NIM microservices, Nvidia NeMo, and agentic AI frameworks from leading providers. The new Nvidia AI blueprints for building agentic AI applications are poised to help enterprises everywhere automate work. Jensen Huang, CEO of Nvidia, made the announcement as part of his CES 2025 opening keynote. With the blueprints, developers can build and deploy custom AI agents that can reason, plan and take action to quickly analyze large quantities of data, including summarizing and distilling real-time insights from video, PDFs and images. CrewAI, Daily, LangChain, LlamaIndex and Weights & Biases are among leading providers of agentic AI orchestration and management tools that have worked with Nvidia to build blueprints that integrate the Nvidia AI Enterprise software platform, including Nvidia NIM microservices and Nvidia NeMo, with their platforms. These five blueprints — comprising a new category of partner blueprints for agentic AI — provide the building blocks for developers to create the next wave of AI applications that will transform every industry. In addition to the partner blueprints, Nvidia is introducing its own new PDF-to-podcast AI blueprint, and another to build AI agents for video search and summarization. These are joined by four additional Nvidia Omniverse blueprints that make it easier for developers to build simulation-ready digital twins for physical AI. Huang noted in his keynote that every programmer will need agents to create code to keep up. To help enterprises rapidly take AI agents into production, Accenture is announcing AI Refinery for Industry built with Nvidia AI Enterprise, including Nvidia NeMo, Nvidia NIM microservices and AI Blueprints. The AI Refinery for Industry solutions — powered by Accenture AI Refinery with Nvidia — can help enterprises rapidly launch agentic AI across fields like automotive, technology, manufacturing and consumer goods. AI agents are a thing. Agentic AI represents the next wave in the evolution of generative AI. It enables applications to move beyond simple chatbot interactions and tackle complex, multi-step problems through sophisticated reasoning and planning. As explained in Huang’s CES keynote, enterprise AI agents will become a centerpiece of AI factories that generate tokens to create unprecedented intelligence and productivity across industries. Agentic AI orchestration is a sophisticated system designed to manage, monitor and coordinate multiple AI agents working together — key to developing reliable enterprise agentic AI systems. The agentic AI orchestration layer from Nvidia partners provides the glue needed for AI agents to effectively work together. The new partner blueprints, now available from agentic AI orchestration leaders, offer integrations with Nvidia AI Enterprise software, including NIM microservices and Nvidia NeMo Retriever, to boost retrieval accuracy and reduce latency. For example:● CrewAI is using new Llama 3.3 70B Nvidia NIM microservices and the Nvidia NeMo Retriever embedding NIM microservice for its blueprint for code documentation for software development. The blueprint helps ensure code repositories remain comprehensive and easy to navigate. ● Daily’s voice agent blueprint, powered by the company’s open-source Pipecat framework, uses the Nvidia Riva automatic speech recognition and text-to-speech NIM microservice, along with the Llama 3.3 70B NIM microservice, to achieve real-time conversational AI. ● LangChain is adding Llama 3.3 70B Nvidia NIM microservices to its structured report generation blueprint. Built on LangGraph, the blueprint allows users to define a topic and specify an outline to guide an agent in searching the web for relevant information, so it can return a report in the requested format. ● LlamaIndex’s document research assistant for blog creation blueprint harnesses Nvidia NIM microservices and NeMo Retriever to help content creators produce high-quality blogs. It can tap into agentic-driven retrieval-augumented generation with NeMo Retriever to automatically research, outline and generate compelling content with source attribution. ● Weights & Biases is adding its W&B Weave capability to the AI blueprint for AI virtual assistants, which features the Llama 3.1 70B NIM microservice. The blueprint can streamline the process of debugging, evaluating, iterating and tracking production performance and collecting human feedback to support seamless integration and faster iterations for building and deploying agentic AI applications. Summarize many, complex PDFs while keeping proprietary data secure The age of Agentic AI is here. With trillions of PDF files — from financial reports to technical research papers — generated everyyear, it’s a constant challenge to stay up to date with information. Nvidia’s PDF-to-podcast AI blueprint provides a recipe developers can use to turn multiple long and complex PDFs into AI-generated readouts that can help professionals, students and researchers efficiently learn about virtually any topic and quickly understand key takeaways. The blueprint — built on NIM microservices and text-to-speech models — allows developers to build applications that extract images, tables and text from PDFs, and convert the data into easily digestible audio content, all while keeping data secure. For example, developers can build AI agents that can understand context, identify key points and generate a concise summary as a monologue or a conversation-style podcast, narrated in a natural voice. This offers users an engaging, time-efficient way to absorb information at their desired speed. Test, prototype and run agentic AI blueprints in one click Nvidia blueprints are aimed at empowering the world’s more than 25 million software developers to easily integrate AI into their applications. These blueprints simplify the process of building and deploying agentic AI applications, making advanced AI integration more accessible than ever. With just a single click, developers can now build and run the new agentic AI blueprints as Nvidia Launchables. These Launchables provide on-demand access to developer environments with predefined configurations, enabling quick workflow setup. Since they contain all necessary components for development, Launchables support consistent and reproducible setups without the need for manual configuration or overhead — streamlining the entire development process, from prototyping to deployment. Enterprises can also

A Non-Sufficient Funds (NSF) or returned item fee is what a bank charges you when it declines a payment due to insufficient funds in your account. If you try to make a payment, such as writing a check without having enough of a balance, the bank will reject the transaction and charge a fee. NSF fees are especially important for businesses to understand, as they can strain cash flow, impact vendor relationships, and even risk account closures if multiple NSF charges accumulate. How returned item fees work Returned item fees kick in when a bank returns a transaction because there aren’t enough funds in your account to cover it. This can happen with checks, electronic payments, or any transactions requiring a balance check. If your bank doesn’t offer overdraft protection, any transaction without the required funds triggers this fee. These fees are particularly troublesome for businesses since failed transactions can lead to unpaid bills, strained relationships with suppliers, and potential service interruptions. If a vendor doesn’t receive payment due to an NSF fee, they might hesitate to work with the business in the future altogether. Returned item fee example To understand returned item fees better, consider the following example. Imagine you’re a small business owner writing a $1,000 check to pay a supplier, but your account balance is only $800. When the supplier deposits the check, the bank refuses it because there’s not enough in your account to cover the payment. This rejection triggers an NSF fee. In addition to the NSF fee, the supplier may also impose a returned check fee on you, further compounding the costs and penalties you can expect. For businesses, these fees are more than just small inconveniences. They represent costs that can directly impact cash flow. If a business accidentally issues several checks without sufficient funds, it could incur multiple NSF fees in a single day, leading to a substantial financial setback. These incidents can strain cash reserves and create budgeting challenges. Managing and monitoring your account balances carefully becomes essential for minimizing the risk of such fees. Returned item fee vs overdraft fee NSF and overdraft fees both relate to insufficient funds, but they differ in handling. A returned item fee happens when the bank declines the payment altogether. In contrast, an overdraft fee applies when the bank allows a transaction to go through, even if it overdraws your account, temporarily covering the cost. How much do returned item fees cost? The average returned item fee now hovers around $20 per incident. The exact cost will vary at each bank, but these fees can quickly add up if multiple payments are rejected in a day. For instance, some banks charge multiple fees for each NSF item presented on the same day, which can turn a minor oversight into a major expense for you and your business. When multiple payments fail, the business faces the bank’s fees and the risk of penalties from vendors or suppliers who expect timely payments. In some cases, vendors may charge their own returned item fees, adding further costs. Businesses should carefully review bank policies on NSF fees and explore ways to minimize them, as these fees can represent an unnecessary drain on resources. Impacts of returned item fees on your business Financial strain: With fees compounding, they can weigh down your budget, especially if your business faces multiple rejected payments. Damaged relationships: Regularly missing payments can harm your reputation with vendors and suppliers, who might refuse future business. If vendors experience frequent payment issues, they may demand cash payments or refuse future business altogether, which can disrupt business growth. Account closure risks: Repeated NSF fees could lead your bank to close your account, which could also affect your credit and lead to future difficulties opening accounts. Banks can strain your business efforts by closing key accounts for repeated NSF activity. More Banking Coverage Steps to avoid returned item fees In addition to reviewing the policies of a bank when opening a bank account or choosing a bank for your business, there are certain steps you can take to avoid the implications of RSFs. Monitor your account balances: Regularly check your balance to ensure funds are available before issuing payments. Set up balance alerts: Many banks allow account alerts for low balances, which can help you avoid bounced checks. Maintain a buffer: Keeping a minimum balance threshold is an effective safeguard. Consider overdraft protection: While it often includes fees, it can prevent returned item fees by covering small shortfalls. What to do if you write a bad check If you inadvertently write a check without enough funds, don’t panic. Here’s what you can do: Notify the payee: Inform the recipient about the situation and arrange an alternative payment. Clear NSF fees: Pay off any returned item fees as soon as possible to avoid further penalties. Request a waiver: If this is your first NSF incident and your account is in good standing, some banks may waive the fee. Conclusion Returned item fees, though small individually, can add up quickly, especially for businesses where cash flow is crucial. They represent a challenge to financial stability but are avoidable with the right strategies. By keeping close tabs on account balances, setting up alerts, and considering overdraft options, you can prevent these fees and maintain better control over your finances. Avoiding NSF fees helps preserve your business’s reputation, ensures timely vendor payments, and strengthens long-term financial health. Frequently asked questions Does an NSF affect your credit? Not directly, but if unpaid balances are sent to collections, your credit score can take a hit. This can impact your ability to gain future loans and credit extensions, further harming your business efforts. Can you get the NSF return fee back? Some banks may waive the fee if it’s your first offense and you have a solid account history. This is not a guarantee though and most banks are strict on enforcing their NSF procedures. Why did your bank charge an NSF fee on

By Jennifer Frank, Matthew Dunn and John Griem ( January 9, 2025, 5:46 PM EST) — The “right-to-repair” movement has been responsible for various states’ laws aimed at making it easier for independent repair shops and individual consumers to repair digital devices, among other electronics and components, by requiring that manufacturers provide diagnostic and repair information, parts and tools to independent repair shops, third-party providers and consumers…. Law360 is on it, so you are, too. A Law360 subscription puts you at the center of fast-moving legal issues, trends and developments so you can act with speed and confidence. Over 200 articles are published daily across more than 60 topics, industries, practice areas and jurisdictions. A Law360 subscription includes features such as Daily newsletters Expert analysis Mobile app Advanced search Judge information Real-time alerts 450K+ searchable archived articles And more! Experience Law360 today with a free 7-day trial. source

今年是港人集體回憶的本地漫畫珍藏《風雲》35週年紀念，香港舞蹈團再次與《風雲》原作者馬榮成，把經典武俠舞劇《風雲》重現舞台。本地餐飲股譚仔(02217)冠名贊助港漫武俠舞劇《風雲》外，即日起更推出期間限定的「雲腿湯米線系列」及多款舞劇《風雲》獨家主題精品，並推出新春限定的「迎春接福過橋米線套餐」。 譚仔全線即日起推出期間限定的「雲腿湯米線配雞卷．風乾火腿」米線，以傳統粵饌菜膽雲腿湯，配金門火腿、西班牙風乾火腿及雞卷。而88元的迎春接福過橋米線套餐，內有15款食材包括西生菜、小棠菜、巴馬火腿、金門火腿、牛肉、雞肉、豬肉、豬膶、冬菇、螺片、豆卜、燻蹄、竹笙、蟲草花及芽菜，寓意新一年豐衣足食，套餐包一杯熱飲。 Pop-up Store 收益貢獻培育香港藝術人才 即日至1月31日期間，兩大傳承聯乘主題店「譚仔 x香港舞蹈團 x港漫武俠舞劇《風雲》」及「譚仔x Grandma’s Scone」進駐尖沙咀星光大道。除了發售多款招牌譚仔小食，如土匪雞翼、四川青椒皮蛋及紅油豬耳外，備受網民追棒的「嫲」辣鬆餅亦會載譽歸來，包括「蒜泥白肉原味鬆餅」、「四川青椒皮蛋芫茜鬆餅」和「酸菜炸醬麻辣5小辣鬆餅」。Pop-up Store有多個精心設計的打卡拍照位，更有多款港漫武俠舞劇《風雲》型格又實用的獨家精品，及譚仔 x Grandma’s Scone精品發售，所有周邊商品收益將予以培育香港藝術人才，舉辦港漫設計比賽招募年輕藝術家，藉此交流及傳承港漫藝術。 香港舞蹈團【舞動《風雲》: 港漫 ╳ 舞蹈 ╳ 光影體驗】 啟動 另外，承載著港人集體回憶的本地經典漫畫《風雲》，跟香港舞蹈團（舞團）結緣於2014年，相隔十年再次攜手原作者，著名漫畫家馬榮成先生載譽重演港漫武俠舞劇《風雲》及舉行糅合舞蹈與藝術科技的大型戶外盛事【舞動《風雲》: 港漫 ╳ 舞蹈 ╳ 光影體驗】。是次盛事於1月11日假 香港文化中心露天廣場進行啟動禮，獲香港特別行政區行政長官夫人 李林麗嬋女士、文化體育及旅遊局局長 羅淑佩女士，JP、中央政府駐港聯絡辦宣文部副部長林枬先生親臨作主禮嘉賓，為盛事揭開序幕。 李林麗嬋表示：「很高興出席香港舞蹈團的舞動《風雲》揭幕儀式，與大家一起走入奇幻的武俠世界。香港舞蹈團與著名漫畫家馬榮成先生相隔逾十年後再次合作，聯同本地頂尖科技藝術團隊，精心糅合舞蹈、漫畫及科技，在香港的文化地標及燦爛的維港夜色下，呈現多姿彩的視覺盛宴，為各位帶來不一樣的沉浸式活動體驗，展現香港獨有的魅力。」 行政長官夫人同時祝賀香港舞蹈團的兩個項目入選「國家藝術基金」，憑荷花獎得獎作品《靜聽松風》，及以李小龍的武術精神為主題的原創大型舞蹈作品《武道》，均獲選國家藝術基金的大型舞台劇目和小型劇目項目，標誌著國家對香港文化藝術界優秀作品的肯定及支持。  香港舞蹈團董事局主席曾其鞏先生,MH表示：「是次大型戶外盛事經過一年多的籌備，現得以成功圓滿舉行，我們衷心感謝香港特區政府、文化體育及旅遊局的鼎力支持，原作者馬榮成先生對舞團一直以來的信任。同時，感謝來自商界的熱心參與，包括舞劇《風雲》的冠名贊助譚仔雲南米線、節目贊助中原地產，以及其他支持單位，天佑銀河慈善基金會、油尖旺民政事務處，DBIS、香港演藝學院、以及香港恒生大學學生的參與。舞團將秉承《文藝創意產業發展藍圖》的發展方向，繼續努力創作優秀的舞蹈作品，積極拓展與商界建立互惠互利的合作，鼓勵更多市民關注及參與文化藝術，吸引更多旅客來港體驗多元文化盛事，讓舞蹈藝術在香港這片土地持續蓬勃發展。」 「舞動《風雲》：港漫 ✕ 舞蹈 ✕ 光影體驗」 盛事後接續登場的足本舞台版《風雲》港漫武俠舞劇，將於2月14至16日假西九文化區戲曲中心公演共五場。門票現正於城市售票網公開發售。 LinkedIn Email Facebook Twitter WhatsApp source

German startup Vay plans to expand its “teledriving” fleet in Las Vegas to 100 electric vehicles — and you could get a job steering the cars.    Vay first launched the service last year, with just two Kia eNiros. It’s fleet has since grown to 30 EVs, which have completed 6000 rides so far.  When you open the Vay app and request a ride, a remote operator drives an electric vehicle to collect you. You then get behind the empty driver’s seat and set off toward your destination.  Once you’re done using the EV, you apply the handbrake, get out, and leave it there in the street, no parking necessary. The teledriver regains control and drives on to the next client.   The 💜 of EU tech The latest rumblings from the EU tech scene, a story from our wise ol’ founder Boris, and some questionable AI art. It’s free, every week, in your inbox. Sign up now! Teledrivers control the vehicles remotely from a purpose-built station equipped with a driver’s seat, steering wheel, pedals, and three monitors providing visibility in front of the car and to its side. The perfect job for gamers? Road traffic sounds, such as emergency vehicles and other warning signals, are transmitted via microphones to the teledriver’s headphones. This operator could technically be sitting on the other side of the world. Most will be nearby at one of Vay’s teledriving centres. As it expands, Vay is looking for more teledrivers to whisk empty cars around Sin City. According to a recent job listing on the company website, here’s how you could land the gig:     You like to drive (and consider yourself a safe and responsible driver). Safety first, second, and third! You are happy working late shifts (afternoons and evenings) as this is what we need for this role. You have a US driving license, clean driving record, and at least two years of driving experience with Uber, Lyft, taxi or similar. You can pass a drug test (including THC). You’re into gaming (or at least super familiar with technology). You are organised and well-structured. You are resilient and have a troubleshooting mindset. You have Google Workplace knowledge (e.g., G-Docs, Sheets, and/or Slides). You are interested in autonomous driving and mobility. If you get the job, you’ll have to pass through Vay’s Remote Driving Academy. The boot camp prepares remote drivers for professional teledriving on public streets and trains them in defensive driving techniques.    Vay bills teledriving as a midway point between conventional cars and autonomous vehicles, which are proving much more difficult to implement than first thought. The company is also making its first foray into remote-controlled trucking.   For drivers, sitting in an office behind a screen is perhaps more comfortable — and definitely safer — than sitting behind the wheel of an actual car or truck.  For customers, it could be a cheaper and more convenient alternative to traditional car-sharing. Vay says the service costs half as much as an Uber. Customers in Vegas pay $0.30 per minute when driving and $0.03 per minute for stopovers. There’s no minimum length or distance and rentals are available for up to 12 hours. For operators of short-term car rental or sharing services, Var claims it can double the amount of time vehicles are in use, boosting revenues.    Vay is the only company ever to have tested a driverless vehicle on public roads in both Europe and the US. Las Vegas was the first city to green-light commercial operations, and Vay hopes to use it as a springboard into the wider US market. Vay also has big plans for Europe — starting with its home country. The company is currently in discussions with German authorities about a domestic rollout.       source

By Tom Paskowitz, Colleen Kenney and Matt Jackson ( January 9, 2025, 5:06 PM EST) — This article is part of a quarterly column analyzing the most notable e-discovery developments from the previous three months. This installment takes a closer look at recent decisions involving the standard principles of proportionality as they apply to the selection of custodians…. Law360 is on it, so you are, too. A Law360 subscription puts you at the center of fast-moving legal issues, trends and developments so you can act with speed and confidence. Over 200 articles are published daily across more than 60 topics, industries, practice areas and jurisdictions. A Law360 subscription includes features such as Daily newsletters Expert analysis Mobile app Advanced search Judge information Real-time alerts 450K+ searchable archived articles And more! Experience Law360 today with a free 7-day trial. source

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More As large language models (LLMs) continue to improve at coding, the benchmarks used to evaluate their performance are steadily becoming less useful. That’s because though many LLMs have similar high scores on these benchmarks, understanding which ones to use on specific software development projects and enterprises can be difficult. A new paper by Yale University and Tsinghua University presents a novel method to test the ability of models to tackle “self-invoking code generation” problems that require reasoning, generating code, and reusing existing code in problem-solving. Self-invoking code generation is much more similar to realistic programming scenarios than benchmark tests are, and it provides a better understanding of current LLMs’ ability to solve real-world coding problems. Self-invoking code generation Two popular benchmarks used to evaluate the coding abilities of LLMs are HumanEval and MBPP (Mostly Basic Python Problems). These are datasets of handcrafted problems that require the model to write code for simple tasks.  However, these benchmarks only cover a subset of the challenges software developers face in the real world. In practical scenarios, software developers don’t just write new code — they must also understand and reuse existing code and create reusable components to solve complex problems. “The ability to understand and subsequently leverage one’s own generated code, [in other words] self-invoking code generation, plays an important role for LLMs to leverage their reasoning capabilities to code generation that current benchmarks fail to capture,” the researchers write. To test the ability of LLMs in self-invoking code generation, the researchers created two new benchmarks, HumanEval Pro and MBPP Pro, which extend the existing datasets. Each problem in HumanEval Pro and MBPP Pro builds on top of an existing example in the original dataset and introduces additional elements that require the model to solve the base problem and invoke that solution to solve a more complex problem.  Self-invoking code generation (source: arXiv) For example, the original problem can be something simple, like writing a function that replaces all occurrences of a given character in a string with a new character. The extended problem would be to write a function that changes occurrences of multiple characters in a string with their given replacements. This would require the model to write a new function that invokes the previous function it generated in the simple problem.  “This evaluation of self-invoking code generation offers deeper insights into the programming capabilities of LLMs, extending beyond the scope of single-problem code generation,” the researchers write. LLMs perform poorly at self-invoking code generation The researchers tested HumanEval Pro and MBPP Pro on more than 20 open and private models, including GPT-4o, OpenAI o1-mini and Claude 3.5 Sonnet, as well as Qwen, DeepSeek and Codestral series. Their findings show a significant disparity between traditional coding benchmarks and self-invoking code generation tasks. “While frontier LLMs excel at generating individual code snippets, they often struggle to effectively [utilize] their own generated code for solving more complex problems,” the researchers write. For example, with a single generation (pass@1), o1-mini achieves 96.2% on HumanEval but only 76.2% on HumanEval Pro. Another interesting finding is that while instruction fine-tuning provides significant improvements on simple coding tasks, it shows diminishing returns on self-invoking code generation. The researchers note that “current instruction-based fine-tuning approaches are insufficiently effective for more complex self-invoking code generation tasks,” suggesting that we need to rethink how we train base models for coding and reasoning tasks. To help advance research on self-invoking code generation, the researchers propose a technique to automatically repurpose existing coding benchmarks for self-invoking code generation. The approach uses frontier LLMs to generate self-invoking problems based on the original problems. They then generate candidate solutions and verify their correctness by executing the code and running test cases on them. The pipeline minimizes the need for manual code review to help generate more examples with less effort. Automatically generating self-invoking code generation problems (source: arXiv) A complex landscape This new family of benchmarks comes at a time when old coding benchmarks are quickly being conquered by frontier models. Current frontier models such as GPT-4o, o1, and Claude 3.5 Sonnet already have very high scores on HumanEval and MBPP as well as their more advanced versions, HumanEval+ and MBPP+.  At the same time, there are more complex benchmarks such as SWE-Bench, which evaluate models’ capabilities in end-to-end software engineering tasks that require a wide range of skills such as using external libraries and files, and managing DevOps tools. SWE-Bench is a very difficult benchmark and even the most advanced models are showing only modest performance. For example, OpenAI o1 is inconsistent on SWE-Bench Verified. Surprising find: OpenAI’s O1 – reasoning-high only hit 30% on SWE-Bench Verified – far below their 48.9% claim. Even more interesting: Claude achieves 53% in the same framework. Something’s off with O1’s “enhanced reasoning”… ?1/8 pic.twitter.com/ADLXNuKpPP — Alejandro Cuadron (@Alex_Cuadron) January 5, 2025 Self-invoking code generation sits somewhere between the simple benchmarks and SWE-Bench. It helps evaluate a very specific type of reasoning ability: using existing code within a module to tackle complex problems. Self-invoking code benchmarks can prove to be a very practical proxy for the usefulness of LLMs in real-world settings, where human programmers are in control and AI copilots help them accomplish specific coding tasks in the software development process. “HumanEval Pro and MBPP Pro are positioned to serve as valuable benchmarks for code-related evaluations and to inspire future LLM development by shedding light on current model shortcomings and encouraging innovation in training methodologies,” the researchers write. source