Your status page at https://status.aspose.cloud/ is showing that your billing service is having a major outage. Unfortunately, this is a common occurrence and it’s causing our self hosted instances to run in evaluation mode, inserting watermarks in generated files.
Please provide timely updates on when this will be resolved.
Furthermore, I thought a safeguard was placed so that there is a seven day grace period when this happens. The grace period is clearly not implemented correctly in your images for self-hosting.
Requests directly to /v4.0/words/info responds with no AdditionalInfo key, where a nested object is expected to contain the IsMeteredLicensed key. We actively poll this as part of the healthcheck in our system. At the time of writing, the AdditionalInfo key isn’t present.
Looks like the billing service is back up and we’re seeing pod restarts initializing successfully. Would still like to get a report of what happened and what your team is going to do to prevent this from happening again. This isn’t the first time your billing service going down wreaked so much havok.
Billing team hasn’t shared the RCA, once it is done I will share it with you.
This issue was caused by migration from windows based containers to linux, it was fixed and won’t happen again. Also they’re planning to ceph buckets from s3 during this week, but it shouldn’t bother any customer as they already migrated several other services from s3 to ceph and these transitions went smoothly.
Thanks for responding with some details. Any word regarding the grace period? From what I understand, the grace period was added a while back to provide resiliency in cases just like this. Clearly, it didn’t work. Our self-hosted instances were responding with no AdditionalInfo key in the response from /v4.0/words/info. When the billing service was operational again, we first started seeing the AdditnionalInfo key with the IsMeteredLicensed object returning "False". Even after a restart of some pods. The value eventually started showing "True" in subsequent responses.
So from our observation, the grace period logic doesn’t actually exist or it’s broken. We’re running the 26.7 tag.
but it shouldn’t bother any customer as they already migrated several other services from s3 to ceph and these transitions went smoothly.
Was the image migration not tested before going forward in your production environment? I would hope so, but there were clearly issues. What is the basis of the confidence with the storage migration?
As I was told migration was tested in the QA env, but seems to be failed in the production, so before migrate to the ceph they conducted multiple tests across different projects, also they already moved several projects that are not so heavily used, and observation showed that it seems to be stable. Today the status page again marked billing outage, but this time it wasn’t a service problem, it was a cluster fail that were fixed.
According grace period I’m asking them why it didn’t work when service were down.
While waiting for the response re: the grace period, another question about the watermarks:
Is there an API or tool available in the container that we can use to programmatically remove or undo the artifacts that were inserted during the erroneous evaluation mode? At this rate, we need to start looking into more options for recovery the next time your billing service outage , or some other mishap on your end, starts causing us these headaches again.
Yes, there is an API with removing watermarks, but if the license isn’t applied this method won’t remove watermark about license.
Your billing service is having a major outage again right now, according to your status page and our internal alerts. Please, I’m begging you at this point, please solve this problem and make it never happen again.
I’m really sorry that it is happened again, I shared your concerns with the billing team, but I’m afraid I can’t fix it as it is other team that responsible for it.
I’ll do my best to highlight this issue so it would be treated with the maximum priority.
Any updates here? What’s the word on the grace period implementation? Also, I see two other topics were split by @awais.hafeez but those conversations are private. This is very much a payment related issue despite paid invoices for us as well. I believe we are obligated to have the same level of support as any other paying customer. Please respond immediately.
Regarding grace period, I was told that it didn’t work as expected due to instability of Redis service that is used to save usage updates from clients, they do many tests on their side to avoid such situations in future.
This leaves me more confused. Are you telling me that the value of the IsMeteredLicense status also relies on the billing service being up? When the billing service goes down, this means this info is also unavailable? This sounds like it breaks the entire point of the grace period.
Please confirm if what you’re saying is actually accurate. If the grace period relies on your billing service being up, then its reliability is pretty much nonexistent.
Furthermore, I’m still waiting for a response about how you’re going to rectify the problem of the consistent disruption of our business operations while our payment remains in good standing.
@james-ontra
The grace period is 7 days, so once you successfully activated the metered license, it is safe for any issues with billing. So the billing team shared the following
Start the pod.
For every request, check the metered license status using IsMeteredLicensed
Only if IsMeteredLicensed returns false, restart pod
IsMeteredLicensed doesn’t contact the billing server. The further actions are:
We will do additional tests with their version because if they kept their pod opened and followed the guidelines mentioned above, then we have a bug on our side that we are not aware about
Hey there! We’ve started to experience the same issue today at ~07:51UTC. Public key is asposeforcloudfaddb8730d7d41bcb23ae1bbe711eae1
The issue is still ongoing. Here’s how it looks in container logs:
Public Key REDACTED *********, Private Key REDACTED ***********
LicenseInitializer:InitLicense() - Failed to initialize metered license for Aspose.Words
System.InvalidOperationException: Authentication failed.
at w.Cg(Object d)
at w.qu(Object d)
at w.mm(w d, B v)
at w.hg()
at w.Bg(Boolean d)
at w.Cg(Object d)
at w.qu(Object d)
at w.xg()
at w.bE(Object d, UInt32 v)
at w.Bg(Boolean d)
at w.mE(Object[] d, Type[] v, Type[] c, Object[] t)
at w.H3(Stream d, String v, Object[] c, Type[] t, Type[] n, Object[] B)
at w.Au(Stream d, String v, Object[] c)
at w.ku(Stream d, String v, Object[] c)
at Aspose.Words.Metered.SetMeteredKey(String publicKey, String privateKey)
at Aspose.Words.Cloud.LicenseInitializer.InitMeteredLicense(String , String)
LicenseInitializer:InitLicense() - Failed to initialize metered license for Aspose.Barcode
System.InvalidOperationException: Authentication failed.
at d.(Exception )
at d(Object )
at d(d , )
at d.()
at d.(Boolean )
--- End of stack trace from previous location ---
at d.(Exception )
at d(Object )
at d.()
at d.(Object , UInt32)
at d.(Boolean )
at d.(Object[] , Type[], Type[] , Object[] )
at d.(Stream , String, Object[] , Type[] , Type[] , Object[] )
at d(Stream , String, Object[] )
at d.(Stream , String, Object[] )
at Aspose.BarCode.Metered.SetMeteredKey(String publicKey, String privateKey)
at Aspose.Words.Cloud.LicenseInitializer.InitMeteredLicense(String , String)
@jurrris
Did you update a container version or it is the same version? 26.7?
@jurrris
I contacted the billing team, but can’t replicate this on my end, just tested with our test credentials
1 Like