Ayende @ Rahien

Marcus Lager commented on Reviewing Resin: Part III

Fri, 14 Jul 2017 09:41:27 GMT

Multiple values for a field is something I've used very, very frequently in Lucene. It's a feature many aren't even aware of exists in Lucene. I think it's an awsome feature. I simplified the problem space just a tiny bit by excluding it from Resin. I regret that decision a little bit. Going from a stream to a byte array and then back to a stream is just silly coding. It was written in a proof of concept state of mind. Even though I haven't taken the time for proper profiling in a while you can witness the amount of work GC has to put up withwhile writing just by watching task mgr's view of the CPU activity, which is a saw pattern going up and down, up and down. Lucene's CPU graph is smooth. Many allocations hotspots are easily removed. In other places there is a need of a immutable structure, a char array. Very good good pointers again.

Oren Eini commented on Reviewing Resin: Part II

Wed, 12 Jul 2017 13:08:08 GMT

Marcus, You might want to look at what it takes to handle time properly in the `Google TrueTime` paper. It is _complex_ and hard. Instead, you want to use something like `Snowflake` or `SequentialUuid` or something like that. What we did was define a vector clock, so each node has its own incrementing value, and we tell them apart using node id.

Oren Eini commented on Reviewing Resin: Part II

Wed, 12 Jul 2017 13:04:34 GMT

Marcus, `Interlocked.Increment` is the way to go here.

Marcus Lager commented on Reviewing Resin: Part II

Wed, 12 Jul 2017 12:58:58 GMT

To resolve the Tick collision case one could use the same mechanisms as when resolving hash collisions. But then we are back to the other requirement, where Resin needs to be able to live in an abide by the rules of a distributed system. To solve *that* one could synchronise the nodes' machine clocks with a time service. After that, is it still problematic in your view to use ticks as timestamps?

Marcus Lager commented on Reviewing Resin: Part II

Wed, 12 Jul 2017 12:43:33 GMT

The popping of a number should be thread-safe is what I mean, to allow many write threads in parallel. I'm going for Resin to either have support directly in the API for a distributed scenario or for it to be treated as a local DB in the way Elasticsearch uses a local db (Lucene) on all of its nodes.

Stephen Ward commented on Reviewing Resin: Part II

Wed, 12 Jul 2017 12:21:44 GMT

@Marcus only thread-safe? is the db only supposed to be a single-node db?

Marcus Lager commented on Reviewing Resin: Part II

Wed, 12 Jul 2017 12:12:16 GMT

The current scheme around Util.GetNextChronologicalFileId is indeed a big fat hack that will not do at all once pressure is on nor when in a distributed environment. What I want is to pop a number from a monotonically increasing number series in a lock free but thread safe manner. The Ticks has played it's part as a stand-in. What's a really solid alternative?

Marcus Lager commented on Reviewing Resin: Part II

Wed, 12 Jul 2017 10:38:04 GMT

Last time I profiled Lucene ~seven years ago there was nothing but string allocations. That and exceptions. When I was prototyping Resin I therefore figured I could afford to care little about those. Lambda's were a great way to move forward fast, to resolve complicated matters swiftly. I figure now it's time to zoom in on the algorithms and of course also the memory management. Very good pointers.

Oren Eini commented on Reviewing Resin: Part I

Wed, 12 Jul 2017 05:58:13 GMT

Svick, I haven't had the time to play properly with spans. The stack only nature is one of their advantages, but it does complicate things. Might be something like: `bool MoveNext(string data, ref Span token);` And the idea is that you send an empty span the first time, then we use the token span as the state to know where to start on the next call.

Marcus Lager commented on Reviewing Resin: Part I

Wed, 12 Jul 2017 05:27:02 GMT

That compare API call you make, I hadn't seen it before. I think I understand now why you would prefer to work with char buffers.

Svick commented on Reviewing Resin: Part I

Tue, 11 Jul 2017 21:34:27 GMT

> Until .NET Core 2.0 is out, I’m going to skip spans and just use direct tokens How would you design the signature of the method with spans, considering their stack-only limitations?

Oren Eini commented on Reviewing Resin: Part I

Tue, 11 Jul 2017 12:35:46 GMT

Remi, The first version of RavenDB did a LOT of things in a highly unoptimized manner. I was focused on getting things sort of working, I didn't even know what I'm supposed to look at compare to what I'm seeing today.

Oren Eini commented on Reviewing Resin: Part I

Tue, 11 Jul 2017 12:34:51 GMT

Marcus, Here you go: ``` List stopWords = new List(); public bool IsStopWord(string value, int offset, int len) { foreach (var word in stopWords) { if (word.Length != len) continue; var cmp = string.Compare(value, offset, word, 0, len, StringComparison.OrdinalIgnoreCase); if (cmp == 0) return true; } return false; } ```

Rémi BOURGAREL commented on Reviewing Resin: Part I

Tue, 11 Jul 2017 11:39:20 GMT

Did you think about this kind of optimizations when you first build RavenDB (even did you have all the knowledge for seing it) or you just applied the "premature optimization is a root of all devil" and optimized your codebase with the time ?

Marcus Lager commented on Reviewing Resin: Part I

Tue, 11 Jul 2017 11:30:21 GMT

*We have the IsStopword method that is now more complex, because it needs to do the check without allocating a string and while being efficient about it.* Given the following restrictions: - a method with the signature bool IsStopWord(string value, int offset, int len) - a list of stopwords as a IEnumerable of string - new string alloc not allowed ... I see no clear path towards a solid equals method If stopwords were a IEnumerable of char arrays we could do SequenceEqual.

Oren Eini commented on RavenDB 4.0: Securing the keys to the kingdom

Tue, 11 Jul 2017 07:51:43 GMT

Pop, There are a few ways to do that. If you care about security, then you register the certificate in the store, apply permissions on who can get it, etc. If you don't care, you just add the certificate to the application. In this way, we are either exactly the same as passwords / api key (which you'll have to get to your app anyway) or more secure. The idea here is that we leave the policy on how to manage it at the hands of the user.

Pop Catalin commented on RavenDB 4.0: Securing the keys to the kingdom

Tue, 11 Jul 2017 06:21:59 GMT

@Stuart Bale, it's more complicated because you have to manage the certificate. The proper way is to install int in the store, then give access to the private key to the right account then retrieve it in the client and use it. If you simply bundle it with your client that's less security not more, it's a self authenticating client.

Pop Catalin commented on RavenDB 4.0: Securing the keys to the kingdom

Tue, 11 Jul 2017 06:04:23 GMT

[note: this was accidentally removed by admin, reposting it] 1) Easy of use: Highly annoying. Creating a client cert, storing in the client somewhere, updating it, etc ... But then you'll want multiple clients with it's own cert and multiple environments, even more annoying. 2) Acceptability: Good for cluster management but for client users. I don't want to manage client certs ... ever. I've done it personally, but had enough of it, it's counter productive. You actually want users in your system not a list of certificates, if you have a list of certificates you'll need to map them to users somewhere else, and nobody will do it, people will start sharing and using the same certificate. 3) Security: Very high in principle, for clients depends on how good the client stores it's secret. It can be better than TLS + passwords if the users don't share the certificates too much.

Oren Eini commented on RavenDB 4.0: Securing the keys to the kingdom

Tue, 11 Jul 2017 06:02:21 GMT

Pop, 1) Given that the admin can generate the certificate right there in the studio, and that the rest just becomes how to send it to the client app, I'm not seeing the difference. Below is a cert, which you can treat as just a very long password. I'm not sure that I'm following the reasoning here. 2) You generate a certificate per user. You don't need to go to a CA to do that, it is possible to do that in RavenDB directly. We also don't manage just the certificate. We identify the cert using its thumbprint, sure, but we are also giving a human readable name to it and apply permissions on that. So it should be convenient. 3) Yeah, but password management it pretty much out of our hands on the client side. ``` MIIKgQIBAzCCCj0GCSqGSIb3DQEHAaCCCi4EggoqMIIKJjCCBg8GCSqGSIb3 DQEHAaCCBgAEggX8MIIF+DCCBfQGCyqGSIb3DQEMCgECoIIE/jCCBPowHAYK KoZIhvcNAQwBAzAOBAgdGv3rkgtJogICB9AEggTYdP9Bu+zaZGtxKnK/vlUP DfdbIzQcmdNp8tokEBVgyVrtNZ8xjAGGrGWdMoadxQJH3WdANz9Pg7Ghico4 0veAmbiA+dXn3GF9tVN2ZkI+QY8LBLw2EKNy7Vlv+M\n1GsGWQeU3yztfcLt HgfrN7nOoOv4HA3sYCswXa/R3Xliq53XdsxZWhXCjbpTx2zE+an9Acr2smwW 18M66uIIxHEUKJZwUaZiQmEOqf9K3vEkp7UMKyEIvD94b2oTioaq4Dk23h8d UfPcVw5L8qiaSzeJVH4AJacN9F20Y8DjyIRYMoIyZ1gIcbe1MAvSsEoEsSgK cgREckda9fP/nZEMIXfWIBgVE7JbnnVWzso7ZLqhTQ1gjT535UAolCYG7KuJ yQmJjaFSb5vHNkvOx42q179v/s\nBaoUoeXpU6JTGJE9n8gtVTnFicn932B4 tPkaP984bXVcCVB+XjZq83AANbA7/SR46CCuggVZC+8+uzYviGXhINJx61Wk Yx4v5PN6f3K3tyYsI1yUoxTkQFRDaL8v1yh2AYjucwc48HeM2Fg0sVR5gQfe +BBtjB9aBHOcQ4mii6f+mKYjbsVm+r6bBuMbtPjG3zZwK0CadmPSiGOULFYi wp3lRC0p3rpCtRoEx/tME9lOW2ulEWTUa0WpsNzKYF7kgep7wgJOhPsNLVHe DQ10aqjeI2\nospB1korPSC2P34wrUBonQ60HTA6YBGx460hOh+4et/HkncV yarZazvUYilOr4Wb52/evN78lTcBEvSdxUeUk1H4ccSnsfyjKsj+wBQSRdzC 2CIt7kp9jn/XnHLpaE6g5U7njFleZN6VfJoIpzWxWY9DIbPMXLeM8vdPuPqu DY8MdDV61SOkn89w2p7zlhHBY7XJbzMXtaJUAQ87uz6pq8GMAb3jTbSXvVd3 x0gR0oe6VhDjKBl57fEyH+Ym+FsfbyKRtQFI7jxlizg6pngomMn8N9\n0EYY NqhiZ9LJsz7iVhrcgB61IIIejqHWAlwDeEL6F9vqPhrhdlPH/MfrL+skDyUV EkxjqNVKtEr2P35aFXDTGaSCUawfDAf3OqPbsjEqSoRyiJUJunJAACGwriVf 0uVkEoWzpcbba+MW08lQ8jL0gJmRcjt1hlS/4tQJGfiVIgzyf9GxITzeIAoW M+xrQT5ispDrEBox+RA1GlXBQSGzMjqg7qI2lxUSWoz6V5Fdpyq7Zj/cHRo6 kvvPzfWl+q0hjQpRfgWfSm0YU8q9Yg2G9aEkUs\n9mV/r/Uzodvdtqb+q0iZ dVdrOfBKRVqPiNbk+TJ1G4Eav4SwN9BExfU79HYD2QBDWqYl1cDwQO+4/+u7 PiuO7PCePVyZ/UfzB81Z2/vAGSF/Ky20nSMHASp5XIbq32q+jlxH1X30gitA 0FSz0NW+ZwzXsd3V5WUGQA7Ef89yaI918zX6haGNnniXoL2pqQ7ntDqKjgI6 BY00Ywmgx1Edu2XeMfLwy0dTZ2oJyePfAbKOtaqqahG9u1OU5FeH2yvrQfFN FjLSqSbx7DMbHy9ionUn+Y\nGq1ipml2wK4VnkS9J4/f4XptPkESNszoYOQ5 1xoRNyz+qcYrgJULG/qM6iqzhZOh4jKotPm63SbveQ6x1WM7T68FB4MQoMY9 rw5M+B1fOOHgi8rv1lbv+TwVSDGB4jANBgkrBgEEAYI3EQIxADATBgkqhkiG 9w0BCRUxBgQEAQAAADBdBgkqhkiG9w0BCRQxUB5OAHQAZQAtADgAOABjAGMA NgBmAGEAMgAtAGYANAA1ADUALQA0ADAAYgA4AC0AOABlAGIAYQAtAGYANAA0 ADAANQ\nBlADYAYgA0ADYAMgAyMF0GCSsGAQQBgjcRATFQHk4ATQBpAGMAcg BvAHMAbwBmAHQAIABTAG8AZgB0AHcAYQByAGUAIABLAGUAeQAgAFMAdABvAH IAYQBnAGUAIABQAHIAbwB2AGkAZABlAHIwggQPBgkqhkiG9w0BBwagggQAMI ID/AIBADCCA/UGCSqGSIb3DQEHATAcBgoqhkiG9w0BDAEDMA4ECOz1VG/SfZ TGAgIH0ICCA8hSm+FeFKjWp69qGA1dftd3X1x6HfSQjeTvGmce\nJbFE73Jz BOKXZ5HS5CfV4Dt4eLKYRn4QvAdf0aNfYIafZ3LdMcTCShMEHf2j7bk9BsPF mQVTbOsFR6vZvsZj4hwy+gwfGyUITYNY/sSojix0ykKOKSOsrOeWkPnJjLj4 EaUIBhzVB2njqkXy4XikcAGOpqLNTbt/85bCar97q4eF+JJZ7CIxna9/xfYT QIUFzow3xYqRMgh5hP3P8mc44Xn2fL4ta+0v7CtbcLLDdU9iyjxAN87POYlZ H56pv0bS7fj4xO+LFymrIi6grJFpCrRDnz\nVGhegqXlwDRBOiu3+qpngsha xlWu1DYnHEZ2FfFspxbn/xhkMun/m2mJTsfVmDYG6PITkPpE2/kVdVBFSIqO 3dJZbHREXNXmvkdAktVJk6rS4I5scNFQr1s1PhxKfZy1C6n6UNH7rw3jhfas HJ/GhQ3a4oBZs+77IE9aD78LXnEP2250Oc1T93T13UEdkcg0ZYfKm+bzbJEq idW52+FtZcKRkodZoddpRCelifNZmKtS4KafgZete+jWzX9kQ13rCu62iTjp jN/5h0+u4sF5q2+UVC\neb8c/ujczzJ7QpqQwUgLmqs8Ws4qoUrSEQyfcx71 eppLKhNIET9mXaDzKBVD1zth9SoofJtGHrjEcns+TVO1AxY8EyxvTxT2Ou58 yNL+uT/z+eL5kU5roX+2MLTBG/uP2fkOINlZWMyVQkRNLm8UXkELoGOEoFSA fiDFLwBFn9zbolXGo24hICig+/KfkNvQ0TgBq2p54hVbSJI7uqQhAyrpkk8f Og53eiMoG/R2xSRT6R9MalG0y0O08xhyCHVpgvLtq9KdZDKSlR7Wj9occu4r MO\nEYZ2eWUb30z9yF2scUF1nAAO+3pjmrhUVCZiZmqiCu17LCGz7gu3aCHP fTJDzHRIEU6ph71vECwsynV68F9/itVJ0oLoXFG6AP0/+rn2LE1hR1Y87J83 XHRtPt4g1b1Jw0oj0x/31iyboZa/AHd+zb3SoTWiRMkmOZ/reE6ynphaAeAe dL59t9qkoi8/rmiiiYifPkbvElLBQBXVdEHh0Bkj+NT28MHQPDY11v+b0fea TKU9ZumSwW2bZ5wsNPjqApXvM47YuI8uE6LkOZAe/FnYri\nSwFuHyLeI2XO iKBXyAUZwEGCoYgWbofDHhT53roc/EdJQkmKQQbzykOjcpJlh/9fVmKzeypL MHsvlntmFll7/e+jGszXqXvPevGEl6ujb+aDpq25jN3wHa9U2+UTzjA7MB8w BwYFKw4DAhoEFMCxndBPhBe6PMJ1rORQCrjSdz0iBBS+hOAPbc+3+b/71t3C 5g/OlFZw6AICB9A= ```

Stuart Bale commented on RavenDB 4.0: Securing the keys to the kingdom

Mon, 10 Jul 2017 21:27:07 GMT

@Pop - how is creating a user account (with username & password) any different to creating a client certificate? Essentially they both have the same management and distribution issues.

Marcin Lewandowski commented on Bad bugs makes for self assigning issues

Mon, 10 Jul 2017 13:23:24 GMT

@Tobias Zürcher We stick with knockout for few reasons: - it was used in v3.X so there is no need to rewrite everything from scratch - it gives us time to provide better user experience, instead rewriting code (see https://pbs.twimg.com/media/Ce5nYp0W4AAxp5Z.jpg) - each framework has it own + & - , we know knockout very well and it is great to feel comfortable when developing - to be honest each js framework has its own advantages and disadvantages - sometimes rewriting application just to be able to use new fancy framework is just replacing one issues with another.

Pop Catalin commented on Bad bugs makes for self assigning issues

Mon, 10 Jul 2017 08:23:27 GMT

@Tobias Zürcher, I don't know Raven's Team members motivations, however Knockout is an actively developed technology, it's used for [Azure Portal](http://jbeckwith.com/2014/09/20/how-the-azure-portal-works/) and it greatly evolved because of it over the years.

Tobias Zürcher commented on Bad bugs makes for self assigning issues

Fri, 07 Jul 2017 19:21:10 GMT

Hey ayende, offtopic but maybe worth a blog post: why did you stick to knockoutjs for the Raven Manageent Studio?

eqr commented on RavenDB 4.0 on Mac OSX

Wed, 05 Jul 2017 09:26:11 GMT

This is very exiting, but using RavenDb in docker works well for development purposes.

Oren Eini commented on RavenDB 4.0: Unbounded results sets

Fri, 07 Jul 2017 20:49:21 GMT

Pure Krome, Streaming gives you streaming access to the data, both on the server and the client. This is a much more efficient system for large requests. You can ping the notification center endpoint for such things, but nothing in the code, and yes, you can set it globally.

Jdan commented on RavenDB 4.0: Unbounded results sets

Fri, 07 Jul 2017 15:27:39 GMT

Hallelujah!

Pure Krome commented on RavenDB 4.0: Unbounded results sets

Fri, 07 Jul 2017 06:25:39 GMT

Oren, - what about the Query Streaming for unbounded results ref: https://ayende.com/blog/161249/ravendbs-querying-streaming-unbounded-results Does this still exist? if so, what use is this now? - I hardly use the studio. Are there any warnings via the code to make this programatic? - How to set this globally?

Oren Eini commented on RavenDB 4.0: Unbounded results sets

Wed, 05 Jul 2017 18:06:58 GMT

Tobias, Yes, of course. Everything in RavenDB is exposed over REST. For example, see: http://4.live-test.ravendb.net/databases/test2/notification-center/watch

Tobias Zürcher commented on RavenDB 4.0: Unbounded results sets

Wed, 05 Jul 2017 17:43:32 GMT

Hi oren, I probably didnt explain myself well enough. If it's configured to just do warnings and allow unbound resultsets, is there a way to query the server to see if there are any warnings? (Not through the raven mgmt studio) This would allow to add the warnings into our custom health check dashboards/monitoring.

Oren Eini commented on RavenDB 4.0: Unbounded results sets

Wed, 05 Jul 2017 12:56:52 GMT

Ian, Typically the size of the request matters a lot, yes. We also have created an issue to add the same warning for very large requests. Note that the expected use case is that you'll either have this flag set for dev only or have it for all you queries. Note that with this flag, you can say `.Take(int.MaxValue)`, which won't throw, but won't add a limit.