How did Newton find derivative of basic functions before formulating systematic Calculus?

The earliest formulations of something that looks like calculus were not in terms of functions but of curves given by algebraic relations. So if we have $$ y=x^2 $$ and $(y+p,x+q)$ is another point on the curve, we would also have $$ (y+p) = (x+q)^2 $$ and if we multiply these equations out and subtract we get $$ p = 2qx + q^2 $$

Now what happens in the earliest sources is that it is simply postulated, without any backing by theoretical definitions or limits that when $p$ and $q$ are small, we can remove their higher powers, so ignoring the $q^2$ term yields us $$ p = 2qx \qquad\text{or, in other words,}\qquad p:q = 2x:1 $$ which can be used to draw a tangent.

The justification for this procedure was initially just that it worked in practice, but the unexplained ignoring of terms like $q^2$ drew quite a lot of contemporary criticism.

This was in the generation before Newton (chiefly Fermat and Descartes). They could produce painstaking geometric proofs in the Euclidean tradition (which was the gold standard for proofs in those days) that for each of the actual curves they considered what came out of this procedure was the right result -- but they didn't have the definitions and machinery to explain or prove rigorously why the procedure always works.

Newton brought in the new idea of a quantity $x$ that varied with time and its time rate of change $\dot x$, but he was still building on the "ignore higher powers of the increments" method. But he did it masterfully and was able to get more out of it than his predecessors did.

Some of these were likely found through a combination of intuition and geometric reasoning.

Consider $y = x^2$. Notice that the square of side length $x$ has area $y$. Suppose that we change the side length by $dx$. Then what is the change in the area of the rectangle $dy$?

I apologize for the crudely drawn picture, but notice that the change in area that we get is,

$$dy = xdx+xdx+(dx)^2 = 2xdx+(dx)^2 \approx 2xdx$$

for small enough $dx$ (i.e. the little black square is insignificantly small). And thus,

$$\frac{dy}{dx} = \frac{2xdx}{dx} = 2x.$$

Similar reasoning can be used to show that $\frac{d}{dx} (x^3) = 3x^2$ by considering the change in volume of a cube.

EDIT: Figured it was worth mentioned that this same square technique can be used to intuitively see why $\frac{d}{dx}(f(x) \cdot g(x)) = f(x)g'(x)+f'(x)g(x)$.

EDIT 2: After you have the product rule, it is easy to derive the power rule (over the natural numbers) using induction. The base case $\frac{d}{dx}(x^1) = 1$ is obvious. If we assume $\frac{d}{dx}(x^k) = kx^{k-1}$, then, $$\frac{d}{dx}(x^{k+1}) = \frac{d}{dx}(x^{k} \cdot x) = \frac{d}{dx}(x^{k})\cdot x + x^{k} \cdot \frac{d}{dx}(x)$$$$ = kx^{k-1}\cdot x+x^k\cdot 1 = kx^k+x^k = (k+1)x^k.$$

Here is some historical information cited from Oskar Beckers Grundlagen der Mathematik in geschichtlicher Entwicklung (say: Foundations of mathematics in historical development).

The first matured presentation of Newtons (1643 - 1727) method of fluxions which was his name of differential calculus dates from 1670/71. The title was Methodus fluxionum et serierum infinitarum and has been posthumously published in 1736.

Newton started to think about fluxions some years before. Here is a small excerpt from one of his early notes cited from O. Becker [ch. Fluxionsmethode p.147].

• Newton: (from 23.11.1665)

  ...If two bodies $A$ and $B$ move uniformly, one of them from $A$ to $C,E$ and $G$, the other from $B$ to $D,F$ and $H$ along the same line, then the line segments $\overline{AC}$ and $\overline{BD}$, $\overline{CE}$ and $\overline{DF}$, $\overline{EG}$ and $\overline{FH}$ behave in the same manner as their velocities $p$ and $q$ and, if they do not uniformly move, then the infinite small line segments behave in each momentum as their velocities in each momentum.

  

  If the body $A$ with velocity $p$ describes the infinite small distance $o$ in a momentum, then the body $B$ will describe in the same momentum with velocity $q$ the distance $\frac{op}{q}$. Since $p:q=o:\frac{oq}{p}$. So, if the line segments already described within one momentum are $x$ and $y$, the next will be $x+o$ and $y+\frac{oq}{p}$.

  Now I'm allowed, if the equation of the line is represented by \begin{align*} rx+xx-yy=0 \end{align*} to substitute $x+o$ and $y+\frac{oq}{p}$ for $x$ and $y$ in the equation. This is valid, since they both mean in the same way as $x$ and $y$ the travelled distance described by the bodies $A$ and $B$. We obtain: \begin{align*} rx+xo+xx+2xo+oo-yy-\frac{2qoy}{p}-\frac{qqoo}{pp}=0 \end{align*} But, according to the assumption we have: \begin{align*} rx+xx-yy=0 \end{align*} after subtraction: \begin{align*} ro+2xo+oo-\frac{2qoy}{p}-\frac{qqoo}{p}=0 \end{align*} and after division by $o$: \begin{align*} r+2x+o-\frac{2qy}{p}-\frac{oqq}{p}=0 \end{align*}

  Now, the terms which contain $o$ are infinitely smaller than the terms which do not contain $o$. So, after cancelling these terms we obtain: \begin{align*} r+2x-\frac{2qy}{p}=0\qquad\text{or}\qquad pr+2px=2qy \end{align*}

  ...

This is an early example how Newton treated problems of differentiation of functions like $y=x^2$. It indicates the development of differential calculus was driven solely to solve physical problems and calculation with infinitesimals was reasoned by physical evidence.

According to O.Becker it was Newtons teacher Isaac Barrow who inspired him to develop these first ideas. On the other hand intensive studies indicate that especially Blaise Pascal inspired Leibniz in his development of differential calculus methods.

Here was how for $f(x)=x^2$:

$$\frac{f(x+o)-f(x)}{o}=2x+o.$$ Now if $o$ gets small, the quotient gets close to $2x$, so Newton declared that when $o$ "gets to" 0, the quotient "gets to" $2x$.

Note, though, that in Newton's times there was no rigorous formulation of limits. So perhaps he was drawing conclusions somewhat from intuition and experimentation. But also keep in mind that intuition was where calculus stemmed from.

This is not intended as an answer to how Newton, in particular, figured out the derivative of a function like $x \mapsto x^2$ but rather to the spirit of the question; furthermore, rather than $x \mapsto x^2$, we will guess the derivative of a general parabolic curve.

First, consider the curve $f: x \mapsto ax^2 + bx + c$ and assume, for the sake of simplicity, that $a>0$.

Accepting that for each point on the curve there is a single tangent-line, the following question is well-defined:

Given $x$, is there a function corresponding to the slope of the tangent-line at $f(x)$?

Recall from high school algebra that the vertex of $f$ occurs at $x = -b/2a$.

Moreover, sketching tangent-lines while moving to the left of the vertex indicates the slopes become negative and keep getting smaller, whereas moving to the right of the vertex indicates the slopes become positive and keep getting larger.

So, our tangent-line slope function is monotonically increasing as $x$ increases, and has exactly one zero; the simplest example that comes to (my) mind of such a function is a linear function. We could also guess how $-b/2a \mapsto 0$: multiply the input by $2a$, then add $b$; in function form, this is a line defined by $x \mapsto 2ax + b$. (If you've studied Calculus: Our guess looks pretty good!)

Note that $c$ does not appear in our final guess, which is a nice reality check: Adjusting the value of $c$ for our initial curve does not affect the slope of any tangent lines; it just moves the curve up or down on the $y$-axis.

So: The OP asks about finding the derivative of $x \mapsto x^2$, but I happen to believe it is possible to guess (as outlined above) the derivative of $x \mapsto ax^2 + bx + c$. Armed with your guess, you can try concrete values for $a, b, c$ and check points to convince yourself that you might be on to something!

I wrote out the ideas above in a paper from this month's issue of NCTM's Mathematics Teacher:

Dickman, Benjamin. "Looking Back to Support Problem Solving." Mathematics Teacher 110.1 (2016): 54-58. Link (no paywall).

Surely any child with an interest in numbers will notice that in a quadratic sequence like 1, 4, 9, 16, 25, ..., the differences between successive terms form a linear sequence like 3, 5, 7, 9, ...? That doesn't need calculus; but someone with a sophisticated mathematical brain will start to recognise that there's a relationship between the function that generates the first sequence and the function that generates the second, and will develop such thinking to discover the principles of calculus.