Veeam reclassified critical VBR7 bugs into non-critical


A week ago I posted a list of Veeam Backup & Replication 7 bugs that I found critical: http://t.co/PRFzEky7Vq (do read my post to become familiar with my point of view).

Anton Gostev, The guy behind Veeam Backup & Replication, Director of Product Management at Veeam Software (http://goo.gl/VwQZQy) answered me: http://goo.gl/B2Ukcx (do read his answer in full):

Regarding critical bugs: in our definition, any bug that results in unrecoverable backup produced, or production data corruption. Only one such bug is reported on v7 so far (in 1 month), and it has a very small scope. Nevertheless, the hotfix is already available. We document critical bugs in our support KB, and there is a dedicated Veeam forum sticky too.

Any other bugs, we don’t normally differentiate between them. ALL issues our customers come to support are hotfixed (unless there is an easy workaround, or extremely minor). The hotfix is provided to a customer, and included in the patch release that we do once every few months (kind of service pack). At this time, we have about 60-70 change sets included in the upcoming patch, however not all of those are bug fixes – part of those are enhancements in the new functionality based on the first feedback.

I’ve been with Veeam since version 1, and v7 has been the most solid release we’ve ever had (especially considering the amount of new functionality). I guess I can say we disagree on the definition of “total disaster” in the blog post title :) to me, this was the most successful major release in history of Veeam.

Regarding specific items you listed:
1. The issue is not specific to v7. There is no case ID, so I cannot be sure – but this might be one of the issues we are working with Microsoft on right now.
2. Not a product issue. There are no complaints in this thread, rather proper deployment considerations dialog. Faster WAN requires faster hardware involved in WAN acceleration. I do disagree with “useless gizmo” assessment on the feature that saves almost 100x WAN bandwidth, as per numbers OP himself has posted in this topic.
3. Not a product issue. Verification speed is determined by backup storage speed. Performance reported is not poor, but rather expected for a lower end NAS on 1Gb LAN, especially if there are other workloads hitting the same NAS. Faster backup storage would result in faster verification.
4. Not a product issue. The problem looks to be specific to storage used (RDX), no similar reports from users using other backup storage. Moreover, we are using the same functionality internally.
5. Not a product issue. Why: we’ve run a poll among our users before making this decision, the results speak for themselves http://forums.veeam.com/viewtopic.php?f=2&t=13400. Note that the poll was run over 1 year ago, so you can adjust results even further towards more latest Windows versions.

Hope this answers your questions and concerns at least partly.

Now let’s go through his answer together while I comment on some of his (and Veeam’s) assumptions and statements concerning all their clients:

‘Regarding critical bugs: in our definition, any bug that results in unrecoverable backup produced, or production data corruption.’ – I am sorry, but this is total BS and it means that they relabeled critical bugs into non-critical for purposes of PR, I guess? Pardon me, but I, as a client, think that if my backup doesn’t work, I am exposed to risk and this is critical for me, even if Veeam does not consider it risky.

‘to me, this was the most successful major release in history of Veeam.’ – This is not for Veeam to decide, but for its clients. When, in the last screen of Backup Wizard window, I see only the ‘Finish’ button enabled, and ‘Back’ and ‘Cancel’ are grayed out, I wouldn’t call it ‘successful release’, that’s ‘buggy release, but that’s just me. Or, rather, it’s not just me: people on forums are saying ‘Clearly Veeam 7 has issues’ (proof: http://communities.quest.com/message/89849#89849 ). Overall view after numerous discussions on forums and among my colleagues is that it is not ‘most successful’, maybe it is the case for enterprise customers, but I am a simple dude with a handful SMB clients (a typical IT consultant) and I am worse off =(((

‘I do disagree with “useless gizmo” assessment on the feature that saves almost 100x WAN bandwidth,’ – If I do compressed incrementals of several different servers, what WAN acceleration I will get? Probability of hits in cache is almost zero and acceleration will be close to zero. Doing full backups of the same servers to get more hits and nice acceleration numbers is what I call ‘useleless gizmo’. I would like to warn everyone: in your project planning, do not count on achieving any WAN acceleration. If you count on it and will not get it because the data was too dissimilar, you will be in the world of trouble. I would say that this new feature is purely theoretical and probably added for marketing purposes, because 100X you would only get in the lab, and in the real world you gain nothing from it.

‘Not a product issue. The problem looks to be specific to storage used (RDX), no similar reports from users using other backup storage. Moreover, we are using the same functionality internally.” – Awful response that means that Veeam is not going to help me. They never mentioned anywhere that RDX is not supported, while it should be clear that RDX is NOT supported. RDX is quite popular, my clients use it and I need it to work, whose problem it is I don’t care.

‘Why: we’ve run a poll among our users before making this decision’ – 109 users of Veeam forum decided on behalf of all clients?! (poll: http://forums.veeam.com/viewtopic.php?f=2&t=13400) That is a very small number compared to the total client base of Veeam. Adding tape support needs OS drivers for it. Microsoft removed tape support in Windows 2008 R2 NTbackup and there are way less tape drivers for 2008 R2 then for 2003. So all the potential clients with tapes 5-15 years old will not be happy because their tapes work only on 2003 and these drives are not cheap – some cost up to 6-number figures in USD. Maybe someone from forums is ok with this, but I am not and there are many customers like me. I was misled. I spent lots of time and even if I get refund, who will pay me for the time I spent on this?

PS BTW, I’ve never gotten an answer from Veeam on my original question: just how many bugs were there in VBR7 so far? It sure seems like at least couple of hundred by the volume of bug reports on their own forums

About these ads

16 thoughts on “Veeam reclassified critical VBR7 bugs into non-critical

    • The only other issue to be discussed is exactly how much you appreciate my style of blogging, but that is already obvious to me as you left your telephone number here 3 or 4 times.

  1. Hello Jim Long/Dwight Warch/etc.

    “An open, public and honest discussion”, it’s not really the case here… we are not having discussion, instead you are posting invalid statements and have people comment your statements. Then, you expand the comments highlighting things that were never said. An honest discussion? Far from it.

    Now, I’ve done some research on your posts across the internet (thanks Google, you can’t hide anything anymore), and judging on quite a few factors:
    – Trying your best at keeping your identity secret, not willing to give your email address or other contact information to Veeam employees. Yet, claiming you are Veeam partner (meaning, we already have your email, phone etc. anyway). Likewise, you cannot open a single support case anonymously. “I do not give out my email”: I am sorry, but how did you become Veeam partner or customer without giving Veeam employees your email address?
    – Unwilling to call or otherwise contact Veeam yourself to discuss the matter, even anonymously. This is simply not logical if you really do care to resolve your and your customer’s issues.
    – Signing exact same posts with absolutely different names on different forums (Jim Long, Dwight Warch to name a few)
    – Cross posting your blog on one of our competitor’s forums, where it is complete offtopic. This even gives me real clues as to who have hired you.
    Based on this, I am going to assume that you are a hired competitive agent (unless you can prove otherwise, even by posting a single support case ID you have ever opened with Veeam support – that’s pretty safe thing to share, right)?

    Nevertheless, I am still going to address your incorrect claims in picking my comment to your post apart.

    1. Let me quote myself again “any bug that results in unrecoverable backup produced”. If your “backup does not work”, this means you have unrecoverable backup. As per my statement, that would be considered a critical issue in Veeam. You are calling BS on response that you did not read…

    2. “When, in the last screen of Backup Wizard window, I see only the ‘Finish’ button enabled, and ‘Back’ and ‘Cancel’ are grayed out, I wouldn’t call it ‘successful release’”: again, this is not specific to v7, the product worked like this since v1. This is by design. The fact that there is no Back or Cancel ability is explained on the previous step (“once you click Next, settings will be saved to the product configuration”). Settings must be saved within the wizard, because the process may take long time or even fail due to SQL issues. Cancel availability would be misleading, because the settings are already saved. For the same reason, there is no going Back. I must also note, that this is the first complaint on this design in 6 years and 75000 customers. However, we are ready to change any design if there is enough customer demand. Feel free to start a topic on Veeam forums to see if other users support your idea. Most features you see in our product today have started from the request posted on our forums by a user or a partner.

    3. “I would say that this new feature is purely theoretical” did you try it outside test lab, or this is your theory as well? Note that 80x data reduction was taken from the Veeam customer post you yourself have referenced. Anyhow, we don’t promise 100x, we promise “up to 50x”, and I would be surprised to see less than 10-20x on real production environment of decent size, based on the statistics we have been receiving so far. In-house, we are getting about 30-40x faster offsite backup across Atlantic. This is a real production environment of 1000+ people company. So, our claims are definitely not baseless or marketing.

    4a. RDX: “Awful response that means that Veeam is not going to help me”. That is incorrect, we are working with all customers who have opened with support case open on troubleshooting this problem. You claim you’ve ran into this issue, what is your support case ID? We cannot help you only in one case, and that is if you don’t open a support case. We provide fanatical support to our customers and partners, and we always do everything possible on our end to resolve the issue, whatever it is. If needed, we will work Tandberg – we have great, established partnership with this company.

    4b. “They never mentioned anywhere that RDX is not supported, while it should be clear that RDX is NOT supported”: do you realize there is about 1 million of disk based storage, and that it is physically impossible to test all of them?

    5. “109 users of Veeam forum decided on behalf of all clients?!”: I’m really sorry if this comes across rude, but you do need to read up some books on statistical analysis. These are *absolutely random* 109 users, so even if we would poll 1090, or 10900 users, the resulting percentage would not change by more than a few percent at most. Further reading:

    http://en.wikipedia.org/wiki/Sample_size_determination

    http://en.wikipedia.org/wiki/Margin_of_error

    6a. “Microsoft removed … NTbackup and there are way less tape drivers for 2008 R2 then for 2003″ that is incorrect statement. Removal of NTbackup is not equal removing Windows tape API, as vast majority of tape backup vendors still rely on this.

    6b. “I am not, and there are many customers like me” then why is it there is not a single support case or a forum post complaining about this? But let me tell you what, please post the list of tape devices you and your clients use, which had tape drivers for 2003, but do not have those for 2008R2, and we will work with the tape vendor to provide you the drivers. Is this a good deal I am giving you here? ;)

    7. “I’ve never gotten an answer from Veeam on my original question: just how many bugs were there in VBR7 so far?”: actually I did answer this, please read my comment that you quoted above.

    Thank you for reading (or skimming) this far.

    • It is very convenient to claim that I was hired by your competitor instead of focusing on the real issue, right?
      The reason for using aliases on forums and not giving you case IDs is that I don’t want to jeopardize my client’s business.
      Other issues I’ll address later, I am wasting way too much time on this.

      • “It is very convenient to claim that I was hired by your competitor instead of focusing on the real issue, right?”
        Yet again, you are responding to me without reading my response. To the most part, my comment above is focusing on addressing what you call “the real issues”.

        “The reason for … not giving you case IDs is that I don’t want to jeopardize my client’s business.”
        You are jeopardizing your client’s business by not opening a support cases on the issue they are having. To clarify, we’ve searched our support DB yesterday, and found that there are not a single support case open on some of the “real issues” you presented on your blog.

    • Now, on technical side,

      4b. Is this some kind of joke? RDX is a quite popular technology – not one device in a million. I get the impression that Mr.Gostev have never heard of RDX at all, despite claims to be close to Tandberg and other vendors.

      5. Don’t know what kind of Statistics Mr.Gostev is talking about – but definitely not the scientfic kind.

      Users on forum are not a random sample – these people are on the forum for a specific reason. They are either advanced IT consultants and admins – who already migrated from 2003, or people who had issues.

      Both groups fall to extreme sides of Bell Curve (http://en.wikipedia.org/wiki/The_Bell_Curve) of normal distribution of users, and cannot be seen as valid sample group of larger community.

      Moreover, it’s irrelevant, as the original question on forum was not ‘Let’s drop Windows 2003 from Veeam’, but rather ‘Which OS do you use for Veeam B&R?’

      That is like asking ‘What hand you use to write?’ and since 90% use the right hand, let’s chop off all the left ones.

      6a. Again, if anyone is not familiar with the subject: please, go to catalog.update.microsoft.com.

      Search “Tape 2003″ – you get 710 results for Tape Drivers in Windows 2003.

      Search “Tape 2008″ gives 145.

      If this doesn’t mean ‘less drivers for 2008′ – I don’t know what does.

  2. I think it is also fair to share here the SpiceWorks thread that you yourself have started. This provides nice recap of actual user experiences with v7, as well as their thoughts about your recent blog posts.

    http://community.spiceworks.com/topic/382450-veeam-backup-replication-7-total-disaster

    If you want to continue the discussion, let’s continue it on the neutral territory (that SpiceWorks thread will work), because it looks like you’ve put my posts here on pre-moderation (even though I am not posting anonymously), and this is yet another bad sign.

    • Totally agree to share the posts on industry forums. 10x
      ALL of your comments, no matter how critical they are of me, ARE OPEN, it’s just that I do not open them right away as you leave them, because I do have other things to do besides this blog, you know.
      BTW, SpiceWorks crowd seems to be very partial to Veeam, they are like religious zealots, ready to tear apart everyone who says a word against their favorite software, so I do not think that it is a good I idea.

  3. Not true! Each of the issues I covered in the original ‘bug list’ for VBR7 corresponds to a link on YOUR forum and therefore a real situation: http://vbscs.wordpress.com/2013/09/13/veeam-backup-replication-7-total-disaster/
    What are you talking about?
    In your response you dismissed all of them as ‘insignificant’ and ‘not a product issues’ therefore making them hypothetical situations. I had no choice but to address this as overall commentary on your company’s practices and manipulations that paint a whole picture. Afterwards you picked my commentary apart by further formalizing it. This is turning into some useless rhetorical exercise. I am taking a couple days off, but I will return to discuss the technical issues you covered.

  4. Pingback: Veeam Backup & Replication 7 – total disaster | VBSCS

  5. Pingback: Veeam Backup & Replication 7: more critical bugs | VBSCS

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s