Consider all of the natural numbers. It has a beginning, therefore it is bordered, therefore it cannot be infinity.
Nope. It's not really (for some definition of "really"), as you say, "bordered". Perhaps you're already familiar with the trivial demonstration that the even numbers are equinumerous with all numbers: just take 2,4,6,8,... and divide each number by 2, whereby you get 1,2,3,4,... So, you already see where I'm going with this???...
...It's pretty much equally easy to map 1,2,3,4,... into ...-4,-3,-2,-1,0,1,2,3,4,... as it was to map 2,4,6,8,... into 1,2,3,4,... For any natural number n, just map it to n/2 for even n, as above, and to -(n-1)/2 for odd n.
So, 1,2,3,4,... are fundamentally just syntactic symbols. Interpreted one way, the usual way, they're the natural numbers. But interpreted our way above, 1,2,3,4... are just a different set of labels/symbols for the entire set of integers, which aren't "bordered".
Edit
------------
Re @FrankHubeny's comment, implicit in the above answer is a slightly more mathematical elaboration that I circumvented, thinking the details aren't of much interest here. (But I'd also have thought it would've been immediately obvious to Frank, with an ms in math and professional programming background, as per his profile.)
The underlying issue is the syntax-vs-semantics distinction I briefly alluded to. It's not 1,2,3,4,... which is bounded/"bordered", per se, but rather the poset ordering, in this case a total order, 1<2<3<4<... that lets you arrange them in a way that imposes a (more-or-less artificial) "beginning" on an otherwise unordered set. And, what's important here is that the cardinality of that set is independent of any imposed (partial or total) order.
Without the natural-number ordering, 1,2,3,4,... are just a collection of strings, "1","2","3","4",...,"998","999",... Programmers frequently generate such "collections" using Backus-Naur form (bnf), a notation for context-free grammars. Indeed, without the natural-number semantics/order imposed on them, our "1","2","3","4",... strings are just a context-free grammar typically called "numerals", and generated by bnf as illustrated at https://courses.cs.vt.edu/~cs1104/Compilers/Compilers.100.html (see the example towards the bottom of that page). Actually, the lecturer there fails to account for strings like "0123","000123",etc with any number of leading zeros. The usual textbook bnf exercise asks for a grammar that excludes them (which I'll also leave as an "exercise for the reader":).
So the upshot is that, as a collection of elements/objects, the natural numbers 1,2,3,4,... are identical/isomorphic to the strings "1","2","3","4",... And then "1" can't really be called the "first string" (although there's yet another "artificial" ordering called the lexicographic order, essentially alphabetic, where it would be first). What you've ultimately got is a very,very,very... big bucket of strings, with no beginning,middle,end; just a big jumbled-up collection. And you know that collection is countably infinite by an analysis of the underlying grammar that generates it. Then, if you want to impose a semantics and order on that collection, it's your business. But it won't affect the cardinality.