The gray area is a matter of scale and purpose. If you have a small business which serves as specific market it's a good idea to make the site as user and disability friendly as possible. If, however, your market serves segments with a likeliness to have various handicaps or disabilities you should address that with an ADA Compliance review of the site. Also consider the scale and purpose of an organization. Publicly traded large businesses need to concern themselves more than a single location family owned retailer and if someone sues your flower shop because the text was too small or didn't have "image alt tags" causing them distress... That's pretty frivolous and would likely not stand.
If your website is not already ADA compliant, you are automatically missing out on millions of potential customers who cannot access your site due to their disabilities. In fact, there are nearly 50 million people with disabilities in the U.S., which means about 19 percent of this country has a disability. Many of them might be interested in your products or services, but once they arrive at your website, they won't be able to navigate easily enough to buy anything or even contact you, all because your website is only accessible to people without disabilities. Thus, they may move on to your competitors.
The fastest, most certain way to be sure your website is in compliance is to contact a qualified web design agency and have them perform an audit of all your online properties. Make sure you interview the agency thoroughly first, as not all agencies are up to speed on ADA website compliance rules. A qualified web design firm will be able to identify any violations of ADA Website Compliance and outline a plan for updating your online content and properties.
While legal considerations might be your biggest worry, making your site more accessible is simply good customer service. More than 39 million Americans are blind and another 246 million have "low vision," Another one million are deaf in the U.S. Add to that people with mobility issues that prevent them from using their hands and that's a huge portion of the country's buying power.