First, a website does not have to be responsive to be optimized for mobile devices. There are several ways to have a mobile friendly website, Google doesn't care how you do it as long as the site meets their mobile guidelines. Responsive sites are a good way to go because it’s generally the easiest type of site to manage for a business owner. Google still accepts purpose built mobile sites on subdomains as well as dynamic code being served based on the user agent of the visitor.
Next, the algorithm change started rolling out on April 21. It does not affect non-mobile search in any way and it is only one factor in hundreds that Google uses to determine search result placement. Google have already specifically stated that the most relevant results will still get top rankings, mobile friendly or not. They are absolutely not removing sites from their index for not being mobile friendly. Your site will still be in Google come May 1st, but your mobile rankings may be affected if your site is not mobile friendly.
That said, it is a big deal to have a mobile optimized site, but mostly because most sites have a high percentage of mobile traffic. Mobile search already makes of about 50% of all searches so it's a good bet that most sites will have a fairly significant amount of mobile traffic that could be bouncing because the site is too hard to use on mobile.
If you don't know if your site is mobile friendly or not, then it probably isn't, but Google will be happy to tell you for free here - https://www.google.com/webmasters/tools/mobile-friendly/